Data processing method and device, electronic equipment, storage medium and program product
By storing the client's real-time data in incremental packets when the acquisition time is later than the data packet update time, and reading and processing data from both the data packets and incremental packets, the problem of the server being unable to obtain the client's real-time data in a timely manner is solved, thereby improving the real-time performance and reliability of the data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-15
- Publication Date
- 2026-03-17
AI Technical Summary
In existing technologies, the server cannot obtain real-time data generated by the client in a timely manner, resulting in low real-time data storage performance.
When the acquisition time is later than the data packet update time, the client's real-time data is stored in the incremental packet, and data is read from the data packet and the incremental packet for processing. The data packet is adaptively stored by the data server.
It improves the real-time performance and reliability of data, ensuring that real-time data generated by the client is stored and processed in a timely manner, and avoiding data loss.
Smart Images

Figure CN115982180B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and more specifically, to a data processing method and apparatus, electronic equipment, computer-readable storage medium, and computer program product. Background Technology
[0002] During operation, the client generates data, which the server retrieves and stores. Currently, the real-time performance of data storage is low, making it impossible to obtain real-time data generated by the client from the server's stored data in a timely manner. Summary of the Invention
[0003] To address the aforementioned technical problems, embodiments of this application provide a data processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.
[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.
[0005] According to one aspect of the embodiments of this application, a data processing method is provided, including:
[0006] Obtain real-time data from the client;
[0007] If the acquisition time of the real-time data is later than the update time of the stored data packet, the real-time data is stored in an incremental packet. The data packet is obtained from the data server, which is used to adaptively store the data sent by the client into the data packet. The update time of the data packet refers to the time when the data server updates the data packet.
[0008] Data is read from the data packet and the incremental packet, and the read data is processed.
[0009] According to one aspect of the embodiments of this application, a data processing apparatus is provided, the apparatus comprising:
[0010] The acquisition module is configured to acquire real-time data from the client.
[0011] The storage module is configured to store the real-time data in an incremental packet if the acquisition time of the real-time data is later than the update time of the stored data packet. The data packet is obtained from the data server, and the data server is used to adaptively store the data sent by the client into the data packet. The update time of the data packet refers to the time when the data server updates the data packet.
[0012] The processing module is configured to read data from the data packet and the incremental packet, and to process the read data.
[0013] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the data processing method as described above.
[0014] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the data processing method as described above.
[0015] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing methods provided in the various alternative embodiments described above.
[0016] In the technical solution provided by the embodiments of this application, real-time data from the client is acquired. If the acquisition time of the real-time data is later than the update time of the stored data packet, the real-time data is stored in an incremental packet, and data is read from the data packet and the incremental packet, and the read data is processed. The data packet is obtained from the data server, which is used to adaptively store the data sent by the client into the data packet. The update time of the data packet refers to the time when the data server updates the data packet. On the one hand, the real-time data generated by the client can be stored and processed in a timely manner, improving the real-time performance of the data. On the other hand, storing the real-time data generated by the client in the incremental packet ensures that even if the data packet is rolled back, the real-time data generated by the client will not be lost, improving reliability.
[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:
[0019] Figure 1 This is a schematic diagram of one implementation environment involved in this application.
[0020] Figure 2 This is a flowchart illustrating a data processing method in an exemplary embodiment of this application.
[0021] Figure 3 yes Figure 2 The flowchart of step S120 in the illustrated embodiment is shown in an exemplary embodiment.
[0022] Figure 4 yes Figure 2 The flowchart of step S130 in the illustrated embodiment is shown in an exemplary embodiment.
[0023] Figure 5 yes Figure 4 The flowchart of step S132 in the illustrated embodiment is shown in an exemplary embodiment.
[0024] Figure 6 yes Figure 4 The flowchart of step S132 in the illustrated embodiment is shown in an exemplary embodiment.
[0025] Figure 7 yes Figure 6 The flowchart of step S320 in the illustrated embodiment is shown in an exemplary embodiment.
[0026] Figure 8 This is a block diagram illustrating data storage in an exemplary embodiment of this application.
[0027] Figure 9 This is a flowchart illustrating index information reading in an exemplary embodiment of this application.
[0028] Figure 10 This is a flowchart illustrating data reading in an exemplary embodiment of this application.
[0029] Figure 11 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of this application.
[0030] Figure 12 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application. Detailed Implementation
[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0033] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0034] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0035] In related technologies, the server acquires and stores data generated by the client. However, the server typically stores the acquired data periodically into data packets or dynamically adjusts the storage time based on its processing capabilities, resulting in poor data real-time performance and an inability to obtain real-time data generated by the client in a timely manner. Based on this, embodiments of this application propose a data processing method and apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the real-time performance of data storage to a certain extent.
[0036] Please see Figure 1 , Figure 1 This is a schematic diagram of an implementation environment related to this application. The implementation environment includes a client 100, a data server 200, and a data processing device 300, which communicate with each other via wired or wireless networks.
[0037] The client 100 can be installed on any electronic device such as a smartphone, tablet, laptop, or computer. The data server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This is not a limitation. The data processing device 300 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This is not a limitation.
[0038] During operation, client 100 generates data, such as new user information and browsing history; client 100 reports this data to data server 200. Upon receiving data from client 100, data server 200 adaptively stores the data in the data packet. That is, after receiving data from client 100, data server 200 can update the data in the data packet according to certain rules. For example, data server 200 can update the received data in the data packet after a certain amount of data has been received; or, data server 200 can periodically update the received data in the data packet, for example, once a day; or, data server 200 can dynamically adjust the update time based on current processing capacity, shortening the update cycle when processing capacity is high and extending the update cycle when processing capacity is low. It should be noted that the methods by which data server 200 adaptively stores data sent by the client in the data packet include, but are not limited to, the examples described here.
[0039] The data processing device 300 can acquire data packets from the data server 200 and store them. To promptly acquire real-time data generated by the client 100, the data processing device 300 can also acquire real-time data from the client 100. If the acquisition time of the real-time data is later than the update time of the stored data packet, the real-time data is stored in an incremental packet. The update time of the data packet is the time when the data server updates the data packet. This approach reduces data storage volume by ensuring that real-time data is only stored when its acquisition time is later than the update time of the stored data packet. Furthermore, storing real-time data in incremental packets ensures that the real-time data is not lost even if the data packet is rolled back, thus improving data reliability.
[0040] During data processing, the data processing device 300 can read data from data packets and incremental packets and process the read data. This allows it to obtain real-time data from the client in a timely manner, improving the real-time performance of the data.
[0041] In one exemplary embodiment, the data server 200 may further include a data analysis module for analyzing the data received from the client 100, and the data server 200 may store the data into a data packet based on the analysis results.
[0042] In another exemplary embodiment, the data packet may include multiple types of data packets, with different types of data packets corresponding to different tags. These tags may correspond to different attribute parameters, such as time, region, and preferences. The data analysis module can analyze the data received from the client and tag the data based on the analysis results. The data server can store the data into the corresponding data packet based on the tags. The data analysis module includes, but is not limited to, a profiling module, which can analyze the data received from the client to obtain user attributes (including but not limited to user attributes such as region, needs, and preferences) and tag the data based on these user attributes.
[0043] The data processing device 300 also stores various types of data packets, and different types of data packets are configured with corresponding incremental packets. After acquiring real-time data from the client, the data processing device 300 can first determine the data packet whose tag matches the tag contained in the real-time data from the various types of data packets. If the acquisition time of the real-time data is later than the update time of the determined data packet, the real-time data is stored in the incremental packet corresponding to the determined data packet.
[0044] based on Figure 1 The application scenarios shown demonstrate that, by adopting the technical solutions of the embodiments of this application, the real-time performance and reliability of the data have been improved.
[0045] The following details the various implementation details of the technical solutions in the embodiments of this application:
[0046] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment. The method can be applied to... Figure 1 The implementation environment shown, and by Figure 1 The data processing device 300 in the environment of the illustrated embodiment is specifically implemented.
[0047] like Figure 2 As shown, in an exemplary embodiment, the data processing method may include steps S110 to S130, which are described in detail below:
[0048] Step S110: Obtain real-time data from the client.
[0049] It should be noted that a client refers to a program that provides local services to customers. It can be installed on any electronic device such as a smartphone, tablet, laptop, or computer. It can be any type of client, such as a video playback client, a music playback client, a content interaction client, or a web browsing client. Among them, web browsing clients include, but are not limited to, browser clients. This embodiment does not limit the type of client or the electronic device on which it is installed.
[0050] The client's real-time data refers to the data generated in real time during the client's operation. For example, in order to improve the real-time nature of account data, the client's real-time data may include new account data, or it may include at least one of the following: browsing history data, running status data, etc. This embodiment does not limit the type of client's real-time data.
[0051] In this embodiment, real-time data from the client can be acquired. This real-time data can be sent directly from the client to the local end, or it can be forwarded by the client to the local end through an intermediate communication device. This embodiment does not limit the method of acquiring real-time data.
[0052] Step S120: If the acquisition time of real-time data is later than the update time of the stored data packet, then the real-time data is stored in the incremental packet.
[0053] First, it should be noted that the local device stores data packets, which can be stored locally or in the cloud. In this embodiment, the storage method of the local device's data packets is not limited.
[0054] Secondly, the data packets stored on this end are obtained from the data server. The data server is connected to the client and is used to adaptively store the data sent by the client into the data packets. The client's data is constantly being generated, so the data server needs to continuously update the data packets. In order for this end to obtain the latest data packets, the data server can send the updated data packets to this end after updating them. This end stores the data packets obtained from the data server.
[0055] The update time of a data packet is the time when the data server updates the data packet. When the data server sends a data packet to the local end, it can also send the update time of the data packet to the local end so that the local end is aware of the update time of the data packet.
[0056] The local end also has an incremental package for storing data. This incremental package can be stored locally or in the cloud. In this embodiment, the storage method of the incremental package is not limited.
[0057] In this embodiment, in order to obtain real-time data from the client in a timely manner for data processing, after obtaining the real-time data from the client, it is first determined whether the acquisition time of the real-time data is later than the update time of the data packet stored on this end. If so, the real-time data is stored in the incremental packet.
[0058] Step S130: Read data from data packets and incremental packets, and process the read data.
[0059] In this embodiment, when data processing is required, data can be read from data packets and incremental packets, and the read data can be processed to improve the real-time performance of the data.
[0060] Depending on the application scenario, the data being read can be processed in different ways. For example, the data can be used for big data analysis; or it can be used for model training; or, if the data stored in the data packets and incremental packets includes account data, the account data can be read from the data packets and incremental packets, and messages can be pushed to the read account data, thus enabling timely push of messages to new accounts. It should be noted that this embodiment does not limit the method of processing the read data.
[0061] In the solution provided in this embodiment, real-time data from the client is first acquired. If the acquisition time of the real-time data is later than the update time of the stored data packet, the real-time data is stored in an incremental packet. Data is then read from the data packet and the incremental packet, and the read data is processed. The data packet is obtained from the data server, which adaptively stores the data sent by the client into the data packet. The update time of the data packet refers to the time when the data server updates the data packet. In this way, on the one hand, the real-time data generated by the client can be stored and processed in a timely manner, improving the real-time performance of the data; on the other hand, storing the real-time data generated by the client in the incremental packet ensures that even if the data packet is rolled back, the real-time data generated by the client will not be lost, improving reliability.
[0062] In one exemplary embodiment, the data packet includes multiple types of data packets for storing different types of data. The different types of data packets have different tags, which are used to characterize different attributes; for example, they may include at least one attribute representing region (country, province, city, etc.), time, hobbies, etc. After receiving data sent by the client, the data server can analyze the data to determine its attributes and store the data in the corresponding data packet based on these attributes.
[0063] In this embodiment, corresponding incremental packets are set for different types of data packets.
[0064] Please see Figure 3 , Figure 3 To handle situations where data packets include multiple data packet types, and different data packet types have corresponding incremental packets, Figure 2 The flowchart of step S120 in the illustrated embodiment is shown in an exemplary embodiment. Figure 3 As shown, if the acquisition time of real-time data is later than the update time of the stored data packet, the process of storing the real-time data in the incremental packet may include steps S121-S122, which are described in detail below:
[0065] Step S121: Determine from the various types of data packets that the tag matches the tag contained in the real-time data.
[0066] It should be noted that real-time data may contain tags, such as tags representing regions. After acquiring the real-time data, the tags contained in the real-time data can be compared with the tags of each type of data packet in various data packets to determine the data packets whose tags match the tags contained in the real-time data.
[0067] Step S122: If the acquisition time of real-time data is later than the update time of the determined data packet, then the real-time data is stored in the incremental packet corresponding to the determined data packet.
[0068] After identifying the data packet that matches the tag contained in the real-time data, if the acquisition time of the real-time data is later than the update time of the identified data packet, the real-time data will be stored in the incremental packet corresponding to the identified data packet.
[0069] In the solution provided in this embodiment, different types of data packets are divided based on different tags, and corresponding incremental packets are set for different types of data packets. This allows real-time data of different types to be stored in the incremental packets of the corresponding data packets, so that data in different types of data packets and their corresponding incremental packets can be read according to actual needs.
[0070] In one exemplary embodiment, an incremental packet is created every preset period, and data acquired within the same period is stored in the same incremental packet. The preset period can be flexibly set according to actual needs; for example, it could be once a day, once every two days, etc. Alternatively, if the data server periodically updates data packets, the preset period can be the same as the data server's data packet update period. In one example, assuming the data server updates data packets once a day, an incremental packet can be created daily, storing the real-time data acquired before the next update period. Or, if the data server periodically updates data packets, the preset period can be shorter than the data server's data packet update period.
[0071] It should be noted that if a data packet includes multiple types of data packets, and different types of data packets have corresponding incremental packets, then the creation period for the corresponding incremental packets of different types of data packets can be the same or different.
[0072] Under the condition that an incremental package is created every preset period, and the data acquired within the same period is stored in the same incremental package, Figure 2 After step S120, the data processing method further includes: when the next preset period arrives, if the number of created incremental packets reaches a preset number threshold, the data in the incremental packet with the earliest creation time is merged into the data packet, and a new incremental packet is created.
[0073] The preset number threshold is the upper limit of the number of incremental packets. The specific value can be flexibly set according to actual needs. For example, it can be set to one, two, three, etc. If the data packet includes multiple types of data packets, the number threshold of incremental packets corresponding to different types of data packets can be the same or different.
[0074] When the next preset period is reached, if the number of incremental packets already created reaches the preset number, the data in the earliest created incremental packet can be merged into the data packet, and a new incremental packet can be created, so that the number of incremental packets does not exceed the number threshold. In this way, by limiting the number of incremental packets, the efficiency of reading data from the incremental packets can be improved.
[0075] In this embodiment, an incremental packet is created every preset period. Data acquired within the same period is stored in the same incremental packet. This reduces the amount of data in a single incremental packet and categorizes the data according to time. This allows the system to determine which packets need to be retained after the updated data packet is retrieved from the data server, based on the packet's update time. Furthermore, when the next preset period arrives, if the number of created incremental packets reaches a preset threshold, the data from the earliest created incremental packet is merged into the data packet, and a new incremental packet is created. This limits the number of incremental packets while ensuring data integrity, preventing the infinite increase in the number of incremental packets and thus avoiding slow reading speeds.
[0076] Please see Figure 4 , Figure 3 To accommodate situations where both data packets and incremental packets contain multiple data fragments, Figure 2 The flowchart of step S130 in the illustrated embodiment is shown in an exemplary embodiment. Figure 3 As shown, the process of reading data from data packets and incremental packets may include steps S131-S132, which are described in detail below:
[0077] Step S131: Obtain the index information of the data fragments contained in the data packet and the incremental packet respectively.
[0078] In this embodiment, both the data packet and the incremental packet contain multiple data fragments, meaning that distributed storage technology is used to store the data packet and the incremental packet. Distributed storage technology disperses the data across multiple independent devices.
[0079] The index information for data fragments is used to locate the corresponding data fragment. For example, in one example, the index information for a data fragment may include the identification number of that data fragment. In another example, when a data packet includes multiple sub-channels, each sub-channel includes multiple data fragments, and an incremental packet includes multiple sub-channels, each sub-channel includes multiple data fragments, the identification information for the data fragment may also include the identification number of the sub-channel to which the data fragment belongs. It should be noted that the sub-channel is used to characterize the type of the data source client. The client type can be based on the type of its electronic device, such as a mobile phone, computer, or tablet. Alternatively, the client type can also be based on the type of its electronic device and the type of system it runs. The system running on the electronic device includes, but is not limited to, iOS (Apple's mobile operating system) and Android. In another example, when a data packet includes multiple versions, the index information for the data fragment may include the read version number. It should be noted that data packets are constantly updated, and multiple versions of data packets can be stored for each type of data packet. Therefore, when multiple versions of data packets of the same type are included, the index information for the data fragment may also include the read version number to locate which version of the data packet to read.
[0080] Specifically, the index information of the data fragments contained in the data packets and incremental packets can be obtained from the index information contained in the data packets and incremental packets. It should be noted that the data packets contain index information used to locate the corresponding data within the data packets. For example, it may include at least one of the following parameters: total data volume of the data packet, total number of data fragments, size of the data fragments, identification number of each data fragment, identification number of the sub-channel, number of sub-channels, number of data fragments contained in each sub-channel, and identification number of the data fragments contained in each sub-channel. Similarly, the incremental packets also contain index information used to locate the corresponding data within the incremental packets. For example, it may include at least one of the following parameters: total data volume of the incremental packet, total number of data fragments, size of the data fragments, identification number of each data fragment, identification number of the sub-channel, number of sub-channels, number of data fragments contained in each sub-channel, and identification number of the data fragments contained in each sub-channel.
[0081] Step S132: Read data from the corresponding data shard based on the obtained index information.
[0082] After obtaining the index information of the data fragments contained in the data packets and incremental packets, the corresponding data fragments can be found based on the obtained index information, and data can be read from the corresponding data fragments.
[0083] In this embodiment, the index information of the data fragments contained in the data packet and the incremental packet is first obtained, and then the data is read from the corresponding data fragments based on the obtained index information, thereby improving the reading speed.
[0084] Please see Figure 5 , Figure 5 for Figure 4 The flowchart of step S132 in the illustrated embodiment is shown in an exemplary embodiment. Figure 5 As shown, the process of reading data from the corresponding data fragment based on the obtained index information may include steps S210-S220, which are described in detail below:
[0085] Step S210: Generate a read request for each data shard based on the obtained index information.
[0086] In this embodiment, when obtaining the index information of the data fragments contained in the data packets and incremental packets, a read request corresponding to each data fragment can be generated based on the obtained index information, and each read request contains the index information of one data fragment. Specifically, the read request can be generated for each data fragment after obtaining the index information of all data fragments contained in the data packets and incremental packets; alternatively, the read request can be generated based on the obtained index information during the process of obtaining the index information of the data fragments contained in the data packets and incremental packets.
[0087] It should be noted that in step S210, the obtained index information of the data fragments can be directly added to the read request. For example, for a read request corresponding to a data fragment contained in a data packet, the identification number of the data packet to which the data fragment belongs, the identification number of the sub-channel to which it belongs, the identification number of the data fragment itself, and the read version number can be added to the read request. For a read request corresponding to a data fragment contained in an incremental packet, the identification number of the incremental packet to which the data fragment belongs, the identification number of the sub-channel to which it belongs, the identification number of the data fragment itself, and the read version number can be added to the read request.
[0088] Alternatively, if the index information of the data shard includes the identification number of the data shard, step S210, which generates a read request for each data shard based on the obtained index information, may include steps S211-S213, as detailed below:
[0089] Step S211: Serialize the index information contained in the data packet and the index information contained in the increment to use as the read version information.
[0090] The index information contained in the data packet and the index information contained in the increment packet are described above and will not be repeated here. For example, the number of data fragments contained in the data packet, the number of data fragments contained in the increment packet, the read version number, and the period corresponding to the increment packet can be serialized and used as read version information.
[0091] In other embodiments, if there is no incremental packet corresponding to the data packet, the read version number can be directly used as the read version information.
[0092] Step S212: For the data fragments contained in the data packet, generate a read request based on the obtained identification number of the data fragment, the identification number of the data packet, and the read version information.
[0093] Step S213: For the data fragments contained in the incremental packet, modify the identity number of the obtained data fragment, and generate a read request based on the modified identity number, the identity number of the data packet, and the read version information.
[0094] The modified identifier number is calculated as follows: The identifier number of the data fragment is equal to the identifier number of the data fragment in the data packet plus the largest identifier number among the data fragments contained in the data packet. For example, assuming the data packet contains 3 data fragments with identifier numbers 001, 002, and 003 respectively, and the identifier number of a certain data fragment in the incremental packet is 005, then after modification, the identifier number of that data fragment in the incremental packet will be 005 + 003 = 008.
[0095] By using steps S211-S213, it is possible to determine whether the read request corresponds to a data packet or an incremental packet based on the parameters inside the read request, without the need to set additional parameters to distinguish whether the read request corresponds to a data packet or an incremental packet.
[0096] Step S220: Read the data contained in the corresponding data fragment according to the index information contained in the read request.
[0097] The read request contains the index information of the data fragment to be read. The corresponding data fragment can be found based on the index information in the read request, and data can be read from the found data fragment.
[0098] After generating a read request, the generated read request can be added to a message queue. In step S220, the read request is retrieved from the message queue, and the data contained in the corresponding data shard is read according to the index information contained in the read request.
[0099] Given that step S210 includes steps S211-S213, step S220, which involves reading the data contained in the corresponding data fragment based on the index information contained in the read request, may include steps S221-S224, as detailed below:
[0100] Step S221: If the read version information in the read request is a string, then the read version information is deserialized to obtain the maximum identity number and read version number.
[0101] Since the read version information is obtained by serializing the index information contained in the data packet and the incremental packet, deserializing the read version information can retrieve the index information contained in the data packet and the incremental packet, and from this, the maximum identity number and the read version number can be obtained. The maximum identity number is the largest identity number among the identity numbers corresponding to the data fragments contained in the data packet.
[0102] Step S222: If the identity number of the data fragment contained in the read request is less than or equal to the maximum identity number, then read the data in the corresponding data fragment according to the identity number of the data packet, the identity number of the data fragment, and the read version number contained in the read request.
[0103] If the identifier of the data fragment contained in the read request is less than or equal to the maximum identifier, it indicates that the read request is for the data fragment contained in the data packet. Therefore, the identifier of the data packet and the identifier of the data fragment can be obtained directly from the read request, and the data in the corresponding data fragment can be read according to the obtained identifier of the data packet, the identifier of the data fragment and the read version number.
[0104] Step S223: If the identity number of the data fragment contained in the read request is greater than the maximum identity number, then the identity number of the data fragment contained in the read request is restored according to the maximum identity number to obtain the restored identity number of the data fragment.
[0105] If the identifier of the data fragment contained in the read request is greater than the maximum identifier, it indicates that the read request corresponds to the data fragment contained in the incremental packet. Therefore, it is necessary to restore the identifier of the data fragment contained in the read request. The restored identifier of the data fragment is equal to the identifier of the data fragment contained in the read request minus the maximum identifier.
[0106] Step S224: Determine the identity number of the incremental packet based on the identity number of the data packet contained in the read request, and read the data in the corresponding data fragment based on the identity number of the incremental packet and the identity number of the restored data fragment.
[0107] Since the read request contains the identifier of the data packet, the identifier of the data packet can be obtained from the read request. Based on the identifier of the data packet, the identifier of the incremental packet corresponding to the data packet can be obtained. Then, based on the identifier of the incremental packet and the identifier of the restored data fragment, the data in the corresponding data fragment can be read, so as to read the data in the data fragment contained in the incremental packet.
[0108] In this embodiment, a read request corresponding to a data fragment is generated first, and then the data in the corresponding data fragment is read according to the read request. This makes the reading process of each data fragment relatively independent, avoiding the situation where the data reading process is interrupted and the data in the entire data packet needs to be read again.
[0109] Please see Figure 6 , Figure 5 for Figure 4 The flowchart of step S132 in the illustrated embodiment is shown in an exemplary embodiment. Figure 6 As shown, the process of reading data from the corresponding data fragment based on the obtained index information may include steps S310-S320, which are described in detail below:
[0110] Step S310: Read data from the data fragments contained in the data packet according to the obtained index information.
[0111] After obtaining the index information of the data fragments, data can be read from the data fragments contained in the data packet.
[0112] Step S320: After reading all the data in the data fragments contained in the data packet, read the data from the data fragments contained in the incremental packet according to the obtained index information.
[0113] Considering the long data reading time, and the possibility of new data being added to the incremental packet during the data reading process, in this embodiment, after the data in the data fragments contained in the data packet has been read, data is then read from the data fragments contained in the incremental packet based on the obtained index information.
[0114] In some implementations, such as Figure 7 As shown, the process of reading data from the data fragments contained in the incremental packet based on the obtained index information in step S320 may include steps S321-S323, which are described in detail below:
[0115] Step S321: Read data sequentially from multiple data fragments contained in the incremental package according to the obtained index information.
[0116] Based on the obtained index information, data is read sequentially from multiple data shards contained in the incremental package. This can be done by reading one data shard before moving on to the next; alternatively, multiple read processes can be created to read multiple data shards in parallel, with one process reading data from one shard before moving on to the next.
[0117] Step S322: When the last data fragment of the incremental packet is read, obtain the index information of the incremental packet.
[0118] When the last data fragment of the incremental packet is read, in order to determine whether new data has been added to the incremental packet during the data reading process, the index information contained in the incremental packet is obtained.
[0119] Step S323: If, based on the obtained index information of the incremental package, it is determined that new data has been added to the incremental package during the data reading process, then the newly added data in the incremental package is read.
[0120] When the index information contained in the incremental package is obtained, it can be determined whether new data has been added to the incremental package during the data reading process. If so, the newly added data and the data of the last data fragment mentioned above can be read, thereby enabling the reading of the data added during the data reading process and improving the real-time performance of the data.
[0121] Specifically, the total amount of data and the total number of data fragments contained in the obtained incremental packet index information can be used to determine whether new data has been added during the data reading process. For example, in step S131, the total amount of data in the incremental packet can be obtained and compared with the total amount of data in the incremental packet obtained in step S322 to determine whether new data has been added to the incremental packet.
[0122] It should be noted that, Figure 5 , Figure 6 The illustrated technical solution can also be combined, that is, first read the data in the corresponding data fragment according to the read request corresponding to the data packet; after the data in the data fragment contained in the data packet has been read, then read the data in the corresponding data fragment according to the read request corresponding to the incremental packet. Specifically, read requests corresponding to the data fragments contained in the data packet can be generated first and placed in a message queue; then read requests corresponding to the data fragments contained in the incremental packet can be generated and placed in the message queue. When reading data, read requests are retrieved from the message queue in a first-in-first-out manner, and data is read from the corresponding data fragment according to the read request, thus ensuring that the data in the data fragments contained in the data packet is read only after the data in the data fragments contained in the data packet has been read.
[0123] In this embodiment, data is read from the data fragments contained in the data packet according to the obtained index information. After the data in the data fragments contained in the data packet is read, data is read from the data fragments contained in the incremental packet according to the obtained index information, thereby improving the real-time performance of the data.
[0124] In another exemplary embodiment, where the incremental packet includes multiple sub-channels, and each sub-channel contains multiple data fragments, Figure 4 Step S132 in the illustrated embodiment may include: sequentially reading data from multiple data fragments contained in each sub-channel of the incremental packet according to the obtained index information; if the last data fragment of any sub-channel is read, then the index information of the incremental packet is obtained, and the number of data fragments contained in the sub-channel is determined according to the obtained index information; if so, then the data in the last data fragment of the sub-channel and the data in the increased data fragment of the sub-channel are read.
[0125] The following describes in detail another specific application scenario of this application embodiment, in which data packets and incremental packets are used to store account data. In this application scenario, the data processing method includes:
[0126] Step S1.0: The data processing device stores the account data from the client.
[0127] It should be noted that during operation, the client will receive an account registration request triggered by a user. The client will then assign a new account to the user and send the corresponding account data to the data processing device and data server. This account data may include at least one of the following: the identification number of the business to which the account belongs, the account's identification number, and the country, province, and city to which the account belongs.
[0128] The data server contains various types of data packets, each with different tags. For example, based on country, province, and city, there are data packets organized by country, province, and city. After receiving account data, the data server periodically updates the data packets with the received account data. During the update process, the profiling module within the data server processes the account data and determines the corresponding tags. The data server then updates the account data to the appropriate data packets based on these tags. After updating the data packets, the data server sends the updated data packets to the data processing device.
[0129] The data processing device retrieves and stores various types of data packets from the data server, and sets up corresponding incremental packets for different types of data packets. After receiving account data from the client, the data processing device identifies data packets whose tags match the tags contained in the account data from the various types of data packets. If the account data was retrieved later than the update time of the identified data packet, the account data is stored in the incremental packet corresponding to the identified data packet. The update time of the data packet is the time when the data server updates the data packet.
[0130] In some implementations, for each type of data packet, the data processing device can create an incremental packet every other day. At the start of the next cycle, the created incremental packet is merged into the corresponding data packet, and a new incremental packet is created, ensuring that only one incremental packet corresponds to each type of data packet at any given time. The storage structure of the incremental packet can adopt a Redis hash structure, where the key can be packet_packetID_date. It should be noted that Redishash is a mapping table of string type fields and values. The data processing device can be configured with a storage SDK (Software Development Kit). The storage SDK merges the data from the incremental packet into the data packet. Since the storage SDK is a unified read / write interface encapsulated for the underlying storage, it shields the differences in read / write details between different underlying storage systems, reducing read / write complexity and improving read / write speed. For example, see... Figure 8 As shown, the write service writes data to the data packet and the incremental packet. The write service writes data to the data packet through the storage SDK. The incremental data merging service merges the incremental packet redis20200818 (i.e., the incremental packet used to store account data obtained on August 18, 2020) into the data packet through the storage SDK, and creates the incremental packet redis20200819 (i.e., the incremental packet used to store account data obtained on August 19, 2020). In this way, when reading data, the read service only needs to read the data from the data packet and the incremental packet redis20200818.
[0131] Step S2.0: Read data from the data packet and the incremental packet.
[0132] Step S2.0 includes steps S2.1 to S2.3.
[0133] Step S2.1: Read the index information.
[0134] Among them, see Figure 9 As shown, step S2.1 includes:
[0135] 1) Read the index information contained in the data packet. The index information contained in the data packet includes the total number of accounts, the total number of accounts contained in each sub-channel, the total number of data fragments contained in each sub-channel, the read version number, and the identity number of the data fragments contained in each sub-channel.
[0136] 2) Determine if there is an incremental package.
[0137] 3) If so, read the index information contained in the increment.
[0138] The incremental data packet contains index information including the total number of accounts, the total number of accounts in each sub-channel, the total number of data fragments in each sub-channel, the read version number, and the identification number of each data fragment in each sub-channel. It should be noted that if the data packet corresponds to an incremental packet, the index information of the incremental packet is read, and the data packet and the index information contained in the incremental packet are serialized to obtain the read version information. Otherwise, the read version number is directly used as the read version information.
[0139] 4) Return the index information that has been read.
[0140] If there is an incremental packet, the incremental packet and the index information contained in the data packet are returned. The index information contained in the incremental packet can be added to the index information contained in the data packet.
[0141] Step S2.2: Generate read requests corresponding to data shards.
[0142] Specifically, for read requests of data fragments contained in a data packet, the requirements include the data packet's identifier, the sub-channel's identifier, the data fragment's identifier, and read version information. For read requests of data fragments contained in an incremental packet, the requirements include the data packet's identifier, the sub-channel's identifier, the modified data fragment's identifier, and read version information. The modified identifier is defined as: the identifier of the data fragment contained in the incremental packet + the largest identifier among the identifiers of the data fragments contained in the data packet.
[0143] Step S2.3: Read the account data from the corresponding data shard according to the read request.
[0144] The process involves first reading the account data from the data packet, and then reading the account data from the incremental packet.
[0145] See Figure 10 As shown, the reading process is as follows:
[0146] (1) Parse the read request.
[0147] (2) Determine whether the read version information in the read request is empty.
[0148] (3) Obtain the latest version number.
[0149] If the read version information is empty, the latest read version number of the data packet can be obtained based on the identifier information of the data packet contained in the read request.
[0150] (4) Determine whether the version information is a string.
[0151] If the version information is not empty, then determine whether the version information is a string.
[0152] (5) The request corresponding to the data packet.
[0153] If the version information is not a string, it indicates that the read request is for the data fragment contained in the data packet, and that the data packet does not have a corresponding incremental packet.
[0154] (6) Deserialize the read version information.
[0155] If the read version information is a string, it is deserialized to obtain the index information contained in the data packet and increment. In one example, when generating the read version information, the index information of the data packet and increment packet can be serialized according to the JSON (JavaScript Object Notation) string format; during deserialization, the read version information can be deserialized according to the JSON string format. JSON is a lightweight data exchange format.
[0156] (7) Determine whether the ID of the data fragment contained in the request is greater than the maximum ID.
[0157] After deserializing the read version information, the maximum ID (i.e., the maximum identifier) can be obtained, and it can be determined whether the ID of the data fragment contained in the read request is greater than the maximum ID. Specifically, if the ID of the data fragment contained in the request is greater than the maximum ID, then the request corresponds to the data fragment contained in the incremental packet; if the ID of the data fragment contained in the request is less than or equal to the maximum ID, then the request corresponds to the data fragment contained in the data packet.
[0158] (8) Read the data from the data fragments contained in the data packet.
[0159] After obtaining the latest read version number, or determining the read request corresponding to the data packet, or determining that the ID of the data fragment contained in the request is less than or equal to the maximum ID, the data in the data fragment contained in the data packet is read according to the read request. Specifically, the corresponding data fragment can be found in the data packet based on the ID of the data fragment contained in the read request and the read version number, so that data can be read from the found data fragment.
[0160] (9) Restore the ID of the data fragment contained in the request.
[0161] If the ID of a data shard contained in the request is greater than the maximum ID, then the request corresponds to a data shard contained in the incremental packet. Therefore, it is necessary to restore the data shard ID. The restored data shard ID is calculated as: (ID of the data shard contained in the request) - (Maximum ID).
[0162] (10) Determine whether the restored ID is the ID of the last data fragment of the sub-channel.
[0163] After restoring the ID of the data fragment contained in the request, determine whether the restored ID is the ID of the last data fragment of its sub-channel.
[0164] (11) Read the index information again.
[0165] If the restored ID is the ID of the last data fragment of its sub-channel, then the index information contained in the increment is retrieved again.
[0166] (12) Determine whether the number of data fragments contained in the sub-channel has increased.
[0167] After reading the index information again, the index information read again is compared with the index information read in step S2.1 to determine whether the number of data fragments contained in the sub-channel has increased during the data reading process.
[0168] (13) Read the added data shards and the data in the data shards corresponding to the request.
[0169] If the number of data fragments contained in a sub-channel increases, then the newly added data fragments of that sub-channel, as well as the data in the data fragment corresponding to the read request, are read. Specifically, when reading the data in the data fragment corresponding to the read request, the data in the corresponding data fragment can be found from the incremental packet based on the restored ID.
[0170] This allows for the reading of newly added data shards and the data within the data shard corresponding to the request all at once.
[0171] (14) Read the data in the data slice corresponding to the request.
[0172] If the restored ID is not the ID of the last data fragment of the sub-channel, or if the number of data fragments in the sub-channel has not increased, then the data in the data fragment corresponding to the request is read. Specifically, the data in the corresponding data fragment can be retrieved from the incremental packet based on the restored ID.
[0173] (15) Returns the data read.
[0174] After reading the data from the data shard, return the read data.
[0175] It should be noted that the read request may also include information such as the identification number of the sub-channel corresponding to the data fragment.
[0176] The data retrieved can be used to push messages to the retrieved account data or to train models based on the retrieved account data.
[0177] Figure 11 This is a block diagram of a data processing apparatus illustrated in an exemplary embodiment of this application. The apparatus includes: an acquisition module 1101 configured to acquire real-time data from a client; a storage module 1102 configured to store real-time data in an incremental packet if the acquisition time of the real-time data is later than the update time of the stored data packet, wherein the data packet is obtained from a data server, the data server is used to adaptively store data sent by the client into the data packet, and the update time of the data packet refers to the time when the data server updates the data packet; and a processing module 1103 configured to read data from the data packet and the incremental packet, and process the read data.
[0178] In another exemplary embodiment, where the data packet includes multiple types of data packets, and different types of data packets are configured with corresponding incremental packets, the storage module 1102 includes:
[0179] The determination module is configured to identify data packets from multiple types of data packets whose tags match the tags contained in real-time data.
[0180] The storage submodule is configured to store the real-time data in the incremental packet corresponding to the determined data packet if the acquisition time of the real-time data is later than the update time of the determined data packet.
[0181] In another exemplary embodiment, where the data packet and the incremental packet each contain multiple data fragments, the processing module 1103 includes:
[0182] The first information acquisition module is configured to acquire the index information of the data fragments contained in the data packets and incremental packets respectively.
[0183] The read module is configured to read data from the corresponding data shard based on the obtained index information.
[0184] In another exemplary embodiment, the reading module includes:
[0185] The generation module is configured to generate read requests for each data shard based on the obtained index information.
[0186] The first reading submodule is configured to read the data contained in the corresponding data shard based on the index information contained in the reading request.
[0187] In another exemplary embodiment, the reading module includes:
[0188] The second reading submodule is configured to read data from the data fragments contained in the data packet based on the obtained index information.
[0189] The second reading submodule is configured to read data from the data fragments contained in the incremental packet based on the obtained index information after the data in the data fragments contained in the data packet has been read.
[0190] In another exemplary embodiment, the second reading submodule includes:
[0191] The first data reading module is configured to read data sequentially from multiple data fragments contained in the incremental package based on the obtained index information.
[0192] The second information acquisition module is configured to acquire the index information of the incremental packet when the last data fragment of the incremental packet is read.
[0193] The second data reading module is configured to read the newly added data in the incremental package if it is determined, based on the obtained index information of the incremental package, that data has been added to the incremental package during the data reading process.
[0194] It should be noted that the data processing apparatus provided in the above embodiments and the data processing method provided in the above embodiments belong to the same concept. The specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.
[0195] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the data processing methods provided in the above embodiments.
[0196] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.
[0197] It should be noted that, Figure 12 The computer system 1600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0198] like Figure 12As shown, the computer system 1600 includes a Central Processing Unit (CPU) 1601, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1602 or programs loaded from storage portion 1608 into Random Access Memory (RAM) 1603, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1603. The CPU 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. An Input / Output (I / O) interface 1605 is also connected to bus 1604.
[0199] The following components are connected to I / O interface 1605: an input section 1606 including a keyboard, mouse, etc.; an output section 1607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to I / O interface 1605 as needed. Removable media 1611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1610 as needed so that computer programs read from them can be installed into storage section 1608 as needed.
[0200] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1609, and / or installed from removable medium 1611. When the computer program is executed by central processing unit (CPU) 1601, it performs various functions defined in the system of this application.
[0201] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.
[0202] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0203] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.
[0204] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.
[0205] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing methods provided in the various embodiments described above.
[0206] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.
Claims
1. A data processing method, characterized by, The method comprises: acquiring real-time data from a client; if the acquisition time of the real-time data is later than the update time of a stored data packet, storing the real-time data into an incremental packet, wherein the data packet is obtained from a data server, the data server is used for adaptively storing data sent by a client into the data packet, and the update time of the data packet refers to the time at which the data server updates the data packet; reading data from the data packet and the incremental packet, and processing the read data; wherein an incremental packet is created every preset period, and data acquired in the same period is stored into the same incremental packet; after the real-time data is stored into the incremental packet, the method further comprises: when the next preset period arrives, if the number of created incremental packets reaches a preset number threshold, merging data in the incremental packet with the earliest creation time into the data packet, and creating a new incremental packet.
2. The method of claim 1, wherein, The data packet comprises multiple types of data packets, and different types of data packets are provided with corresponding incremental packets; if the acquisition time of the real-time data is later than the update time of the stored data packet, the real-time data is stored into an incremental packet, comprising: determining a data packet from the multiple types of data packets, wherein the label in the real-time data matches the label in the determined data packet; if the acquisition time of the real-time data is later than the update time of the determined data packet, storing the real-time data into the incremental packet corresponding to the determined data packet.
3. The method of claim 1, wherein, The data packet and the incremental packet respectively comprise multiple data shards; reading data from the data packet and the incremental packet comprises: acquiring index information of the data shards contained in the data packet and the incremental packet; reading data from the corresponding data shard based on the acquired index information.
4. The method of claim 3, wherein, The reading data from the corresponding data shard based on the acquired index information comprises: generating a reading request corresponding to each data shard based on the acquired index information; reading data contained in the corresponding data shard according to the index information contained in the reading request.
5. The method of claim 3, wherein, The reading data from the corresponding data shard based on the acquired index information comprises: reading data from the data shard contained in the data packet according to the acquired index information; after the data in the data shard contained in the data packet is read, reading data from the data shard contained in the incremental packet according to the acquired index information.
6. The method of claim 5, wherein, The reading data from the data shard contained in the incremental packet according to the acquired index information comprises: reading data from the multiple data shards contained in the incremental packet in sequence according to the acquired index information; when the last data shard of the incremental packet is read, acquiring index information of the incremental packet; if it is determined that the incremental packet has added data in the data reading process according to the acquired index information of the incremental packet, reading the added data in the incremental packet.
7. A data processing apparatus, characterized by, The device comprises: an acquisition module configured to acquire real-time data from a client; The storage module is configured to store the real-time data into an incremental package if the acquisition time of the real-time data is later than the update time of the stored data package, wherein the data package is obtained from a data server, the data server is used to adaptively store data sent by a client into the data package, and the update time of the data package refers to the time when the data server updates the data package; an incremental package is created every preset period, and data obtained in the same period is stored into the same incremental package; The processing module is configured to read data from the data package and the incremental package, and process the read data; The device is further used to, after storing the real-time data into the incremental package, if the number of created incremental packages reaches a preset number threshold when the next preset period arrives, merge data in the incremental package with the earliest creation time into the data package, and create a new incremental package.
8. An electronic device, comprising: comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the method of any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, having computer readable instructions stored thereon, which, when executed by a processor of a computer, cause the computer to perform the method of any one of claims 1-6.
10. A computer program product comprising computer instructions, characterized in that, the computer instructions, when executed by a processor, implement the method of any one of claims 1-6.
Citation Information
Patent Citations
Data storage method, apparatus and system
CN104967640A
Data deduplication method and device, equipment and medium
CN110457305A
Real-time data label acquisition method and device, computer equipment and storage medium
CN110825820A