Fragmentation processing method, device and equipment for service data packet and storage medium

Abstract

The application discloses a sharding processing method and device for service data packets, equipment and a storage medium. The method comprises the following steps: obtaining a service data packet to be processed; determining a sharding identifier corresponding to a plurality of user information according to the number of the plurality of user information, the number of stored user information and a preset number threshold; writing the plurality of user information into a cache database according to the sharding identifier; and migrating a sharding corresponding to the sharding identifier to a target database when the number of user information in the sharding corresponding to the sharding identifier meets the requirement of the preset number threshold. The user information in the service data packet is stored in two levels based on the sharding idea to improve the reading performance. On this basis, the target database as the final storage can not be limited to using a memory type storage system, can support multiple storage types, and can adopt other types of storage systems which are more cost-saving. The application embodiment can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and Internet of Vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet communication, and in particular to a sharding processing method and device for service data packets, an equipment and a storage medium. BACKGROUND

[0002] With the rapid development of Internet communication technology, the business scenarios faced by Internet products in providing relevant product services are becoming more and more complex. Taking the storage of service data as an example, in order to ensure high read speed, the data is often directly stored in a memory type storage system (such as a machine memory) in the related technology, so as to directly access the system for data reading. However, the cost of the memory type storage system is high, and the above-mentioned solution in the related technology is not applicable to large amount of service data. Therefore, a service data storage solution that takes into account the cost and read speed needs to be provided. SUMMARY

[0003] In order to solve the problems of high storage cost and inapplicability to storage of large amount of service data when the prior art is applied to store service data, the present application provides a sharding processing method and device for service data packets, an equipment and a storage medium:

[0004] According to a first aspect of the present application, a sharding processing method for service data packets is provided, the method comprising:

[0005] obtaining a service data packet to be processed; wherein the service data packet to be processed comprises a plurality of user information indicating a specified data packet identifier, the service data packet to be processed is the content carried by any of a plurality of service requests sent in batches, and the plurality of service requests all indicate the specified data packet identifier;

[0006] determining a sharding identifier corresponding to the plurality of user information according to the number of the plurality of user information, the number of the stored user information and a preset number threshold; wherein the data packet identifier corresponding to the stored user information is the specified data packet identifier, and the preset number threshold indicates an upper limit of the number of user information in a shard provided by a target database providing shard read service;

[0007] writing the plurality of user information into a cache database according to the sharding identifier;

[0008] migrating the shard corresponding to the sharding identifier to the target database when the number of user information in the shard corresponding to the sharding identifier meets the requirement of the preset number threshold.

[0009] According to a second aspect of the present application, a sharding processing device for service data packets is provided, the device comprising:

[0010] The acquisition module is configured to acquire a service data packet to be processed, wherein the service data packet to be processed comprises a plurality of user information indicating a specified data packet identifier, the service data packet to be processed is carried by any one of a plurality of service requests sent in batches, and the plurality of service requests all indicate the specified data packet identifier.

[0011] The determination module is configured to determine a shard identifier corresponding to the plurality of user information according to a quantity of the plurality of user information, a quantity of stored user information, and a preset quantity threshold, wherein the data packet identifier corresponding to the stored user information is the specified data packet identifier, and the preset quantity threshold indicates an upper limit of a quantity of user information in a shard provided by a target database providing shard reading service.

[0012] The cache module is configured to write the plurality of user information into a cache database according to the shard identifier.

[0013] The migration module is configured to migrate a shard corresponding to the shard identifier to the target database when a quantity of user information in the shard corresponding to the shard identifier meets a requirement of the preset quantity threshold.

[0014] According to a third aspect of the present application, an electronic device is provided, which comprises a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to implement the shard processing method for service data packets according to the first aspect.

[0015] According to a fourth aspect of the present application, a computer readable storage medium is provided, the storage medium stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by a processor to implement the shard processing method for service data packets according to the first aspect.

[0016] According to a fifth aspect of the present application, a computer program product or a computer program is provided, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the shard processing method for service data packets according to the first aspect.

[0017] The shard processing method for service data packets, the device, the equipment and the storage medium provided by the present application have the following technical effects:

[0018] The application obtains a to-be-processed service data packet, then determines a shard identifier corresponding to the plurality of user information according to the number of the plurality of user information, the number of the stored user information, and a preset number threshold, writes the plurality of user information into a cache database according to the shard identifier, and migrates a shard corresponding to the shard identifier to a target database when the number of the user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold. The to-be-processed service data packet is content carried by any service request in a plurality of service requests sent in batches, the plurality of service requests indicate a same data packet identifier, and the stored user information corresponds to a data packet identifier same as the plurality of user information. The application is applied to a scenario of sending service requests in batches by a service party, wherein each service request carries a service data packet, and one service data packet includes a plurality of user information. The user information in the service data packet is stored in two levels based on the shard idea. In the face of the situation that the number of user information indicated by different service requests is different, the received user information is integrated in a fixed size by using a preset number threshold indicating a shard size, and the shard meeting the requirement is migrated from the cache to the target database to facilitate the implementation of shard reading service, so that the reading performance can be improved. On this basis, the target database for final storage can not be limited to using a memory type storage system, can support multiple storage types, and can adopt other types of storage systems which are more cost-saving. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart of a shard processing method for a service data packet provided by an embodiment of the present application;

[0022] Figure 3 is a flowchart of determining a specified data packet identifier provided by an embodiment of the present application;

[0023] Figure 4 is a flowchart of determining a shard identifier provided by an embodiment of the present application;

[0024] Figure 5 is a hierarchical architecture diagram of a number packet system provided by an embodiment of the present application;

[0025] Figure 6 is a module architecture diagram of the number package system provided by the embodiment of the present application;

[0026] Figure 7 is a response flowchart diagram for a start instruction provided by the embodiment of the present application;

[0027] Figure 8 is also a flowchart diagram of a fragmentation processing method for a service data package provided by the embodiment of the present application;

[0028] Figure 9 is a response flowchart diagram for an end instruction provided by the embodiment of the present application;

[0029] Figure 10 is also a module architecture diagram of the number package system provided by the embodiment of the present application;

[0030] Figure 11 is a diagram of two-level storage of user information provided by the embodiment of the present application;

[0031] Figure 12 is a diagram of version maintenance of user information provided by the embodiment of the present application;

[0032] Figure 13 is a component block diagram of a fragmentation processing device for a service data package provided by the embodiment of the present application;

[0033] Figure 14 is a structural diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0035] It should be noted that the terms “include” and “have” and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover the inclusion of non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those clearly listed steps or units, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] Before the embodiments of the present application are described in further detail, terms and names involved in the embodiments of the present application are explained, and the terms and names involved in the embodiments of the present application are applicable to the following explanations.

[0037] Database (DB): It can be regarded as an electronic file cabinet, i.e., a place for storing electronic files, and users can perform operations such as adding, querying, updating, and deleting data in the files. The so-called "database" is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs.

[0038] Distributed system: It is a system composed of a group of computer nodes that communicate through a network and coordinate work to complete common tasks.

[0039] Message queue: A container for saving messages in the transmission process of the messages.

[0040] Real-time data sharding: In a distributed system, for some continuously pushed data requests of variable size, an intermediate cache is used to accumulate data shards of fixed size, and then the data shards are written into the final storage.

[0041] SDK (Software Development Kit): Software development kit.

[0042] COS (Cloud Object Service): Object storage service.

[0043] BDB (Berkeley Database): A database.

[0044] Redis (Remote Dictionary Server): Remote dictionary service, a database.

[0045] Kafka: A high-throughput distributed publish / subscribe message system.

[0046] MDB (Microsoft Database): A database.

[0047] APP (Application): Application software.

[0048] Please refer to Figure 1 , Figure 1Fig. 1 is a schematic diagram of an application environment provided by an embodiment of the present application. The application environment can include a client 10 and a server 20. The server 20 stores user information in a service data packet based on a sharding idea in two levels. The service data packet is from a service request received by the server 20. A user can send a data update request to the server 20 through the client 10. The server 20 reads the stored user information according to the received data update request to determine user information adapted to the request and returns to the client 10. The client 10 and the server 20 can be directly or indirectly connected through wired or wireless communication. It should be noted that, Figure 1 This is only an example.

[0049] The client 10 can be an entity device such as a smart phone, a computer (such as a desktop computer, a tablet computer, a notebook computer, etc.), an augmented reality (AR) / virtual reality (VR) device, a digital assistant, a smart wearable device, a smart voice interaction device (such as a smart speaker, etc.), a smart home appliance, a vehicle terminal, etc. or a software such as a computer program running in the entity device. The operating system corresponding to the client can include an Android system, an iOS system (a mobile operating system developed by Apple Inc.), a Linux system, a Microsoft Windows system, etc.

[0050] The server 20 can be a physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The server can include a network communication unit, a processor, a memory, etc. The server can provide background services for the corresponding client.

[0051] In actual applications, the execution subject of the sharding processing scheme for the service data packet provided by the present application can be regarded as a number packet system, as shown in Figures 5-10 As shown in Figure 6 , the data sources of the service data packet to be processed include 1) data uploaded manually through a management station front end; 2) data written to a COS; and 3) data pushed to a number packet pushing service. The data involved in 1) and 2) is pushed to the number packet pushing service through a download service. The scheme provided by the present application is mainly for the data involved in 3), and is also applicable to the data involved in 1) and 2).

[0052] The number package system can be regarded as a distributed system, which is used to store a plurality of continuous data requests with different sizes in a distributed storage system as an intermediate cache, and then write the data with a fixed size into a final storage. The distributed storage system and / or the final storage can be deployed on a plurality of servers. Therefore, the method and device for processing the service data package disclosed in the present application can be used to form a blockchain by using a plurality of servers as nodes in the blockchain.

[0053] The following describes a specific embodiment of a method for processing a service data package, Figure 2 is a flowchart of a method for processing a service data package according to an embodiment of the present application. The method provided in the present application includes more or fewer operation steps based on conventional or non-creative labor. The order of the steps listed in the embodiment is only one of the many execution orders, and does not represent the only execution order. In actual system or product execution, the method can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment). For example, as shown in Figure 2 The method can include the following steps:

[0054] S201: obtaining a service data package to be processed; wherein the service data package to be processed includes a plurality of user information indicating a specified data package identifier, the service data package to be processed is the content carried by any of a plurality of service requests sent in batches, and the plurality of service requests all indicate the specified data package identifier;

[0055] In the embodiment of the present application, the server obtains a to-be-processed service data packet. The to-be-processed service data packet comes from a service request sent by an upstream service party. The upstream service party can send multiple service requests indicating the same service to the server in batches, and the to-be-processed service data packet can come from any one of the multiple service requests. Since the multiple service requests indicate the same service, the multiple service requests indicate the same data packet identifier (packet ID), that is, the indicated data packet identifier and the indicated service have an association relationship. The multiple service requests are sent in batches, and the number of user information in the service data packet carried by the service request sent in each batch can be different, such as 200 pieces of user information carried by service request 1 and 300 pieces of user information carried by service request 2. Here, the to-be-processed service data packet obtained by the server includes multiple pieces of user information indicating a specified data packet identifier. For example, the multiple service requests sent by the upstream service party can indicate a “portrait service”, and the user information in the service data packet carried by the service request can be a user portrait label. Generally, one piece of user information corresponds to one user. It should be noted that the server can receive service requests from at least two upstream service parties, each upstream service party can be responsible for at least two services, and accordingly, each upstream service party can send service requests indicating at least two services to the server. In actual application, the upstream service party and the server can belong to the same background system, and the upstream service party can also be affiliated to a third party. In addition, the “portrait service” described above can be an application for various scenarios of Internet of Vehicles or an application for intelligent transportation, and the “user portrait label” described above can indicate the characteristics of a driver or a passenger.

[0056] In an exemplary embodiment, as shown in Figure 3 Before obtaining the to-be-processed service data packet, the method further includes:

[0057] S301: In response to the received service request, determining a first service identifier, a second service identifier and data version information corresponding to the service request;

[0058] S302: According to the first service identifier and the second service identifier, searching the index database to find whether there is a matching data packet identifier; wherein the index database is used to record the relationship between the data packet identifier and its associated object;

[0059] S303: When there is, taking the matching data packet identifier as the specified data packet identifier, and writing the data version information into a first target associated object; wherein the first target associated object is the associated object of the matching data packet identifier in the index database;

[0060] S304: When there is not, create a corresponding data packet identifier as the specified data packet identifier based on the first service identifier and the second service identifier, and write the first service identifier, the second service identifier and the data version information into a second target association object; wherein the second target association object is an association object of the corresponding data packet identifier in the index database.

[0061] The service request sent by the upstream service provider carries a first service identifier and a second service identifier for describing relevant service information. The server searches the index database (index redis) based on the first service identifier and the second service identifier to find whether there is a matching data packet identifier. The index database is used to record the relationship between the data packet identifier and its association object, such as the relationship between the data packet identifier and the service identifier group (first service identifier-second service identifier) as the association object. When there is a matching data packet identifier, the server takes the matching data packet identifier as the specified data packet identifier, and writes the data version information into the first target association object, which is the association object of the matching data packet identifier in the index database. For example, the first service identifier is appID, and the second service identifier is service label A. If there is a matching data packet identifier, it means that the index redis records the mapping relationship between the packet ID: xxxxxx and the service identifier group: appID-service label A. The packet ID: xxxxxx is the matching data packet identifier and the specified data packet identifier, and the service identifier group is the association object of the data packet identifier. It can be understood that the service identifier group: appID-service label A can be used as a search term to search the packet ID: xxxxxx in the index redis. Correspondingly, the data version information is also written into the above association object.

[0062] When there is no matching data packet identifier, the server creates a corresponding data packet identifier based on the first service identifier and the second service identifier as the specified data packet identifier, and writes the first service identifier, the second service identifier and the data version information into the second target association object, which is the association object of the corresponding data packet identifier in the index database. In combination with the above example, if there is no matching data packet identifier, it means that there is no record of the package ID that has a mapping relationship with the service identifier group: appID-service label A in the index redis. It can be understood that the service identifier group: appID-service label A cannot be used as a search term to search for related package IDs in the index redis. Therefore, a corresponding data packet identifier is created for the service identifier group, and the relationship between the data packet identifier and the service identifier group and the data version information is recorded in the index redis, wherein the service identifier group and the data version information are used as the association object of the data packet identifier. The creation and matching of the data packet identifier help establish a service data packet sharding processing scheme in the dimension of the data packet identifier, which can conveniently convert the service identifier provided by the upstream business party and accurately capture the business precision indicated by the service identifier provided by the upstream business party.

[0063] The upstream business party sends multiple service requests indicating the same business to the server in batches, and these service requests can indicate the same version or different versions. For example, the service requests 100-150 indicating business A sent to the server in batches indicate version V2.2, and the service requests 151-222 indicating business A sent to the server in batches indicate version V2.3. Even if they indicate the same business, when the service requests indicate different versions, recording the data version information can help subsequent incremental sharding identifier allocation operations for business data packets belonging to the same version, which can improve the subsequent sharding integration capability for the business data packets carried by multiple service requests sent in batches.

[0064] The service request sent by the upstream business party carries the first service identifier, the second service identifier, the business data packet and the data version information. After the specified data packet identifier is determined, the specified data packet identifier can be used to represent the business data packet carried by the related service request. Correspondingly, the multiple pieces of user information included in the business data packet to be processed indicate the specified data packet identifier.

[0065] In actual application, the upstream business party sends multiple business requests indicating the same business to the server in batches. For multiple business requests indicating the same version, the upstream business party can send a start instruction to the server before sending these business requests to the server. If the execution subject of the fragmentation processing scheme for the business data packet provided in the present application is regarded as a number packet system, the system includes an index redis, a number packet pushing service, a number packet management service and a number packet writing service. The start instruction aims to inform the number packet system that the business party is about to start pushing data and request the number packet system to allocate a packet ID. Referring to Figure 7 , the start instruction carries an appID (corresponding to the first business identifier described above), a thirdID (corresponding to the second business identifier described above) and data version, and requests a start interface of the number packet pushing service. The number packet pushing service queries the packet ID in the index redis according to the appID+thirdID. 1) If the packet has been pushed before, the packet ID can be found here, and then the index information (such as data version information) of the current push is recorded in the index redis, and the packet ID found is returned to the business party; 2) If it is the first time to push, the packet ID cannot be found here, and then the number packet management service is requested to add a packet ID, and the index information (such as the added packet ID, appID+thirdID and data version information) of the current push is recorded in the index redis, and the added packet ID is returned to the business party. It should be noted that the information recorded in the index redis can be regarded as number packet metadata, and the operation related to information recording can be regarded as an atomic operation.

[0066] The upstream business party requests a packet ID by using the start instruction, and can start sending multiple business requests indicating the same business to the server in batches. The content carried by these business requests can only include business data packets and packet IDs, or can also include data version information, a first business identifier and a second business identifier. Referring to Figure 8 , the write data request (business request) is pushed by the business party through a write data interface, and each write data request carries a certain amount of user information. The business party completes data pushing by calling the write data interface multiple times.

[0067] In an exemplary embodiment, after the business data packet to be processed is obtained, the method further includes the following steps: first, splicing the user identifier carried by the user information, the corresponding specified data packet identifier and the indicated data version information to obtain a feature object corresponding to the user information; then, based on the feature object corresponding to the user information, searching the cache database and the target database for user information matching the feature object; and finally, when there is, removing the user information from the multiple pieces of user information.

[0068] Considering that the upstream business party may send the relevant user information repeatedly for its own reasons, and the upstream business party may cause the relevant user information to be sent repeatedly due to timeout and other reasons when calling the relevant interface to send data to the server, the user information of the to-be-processed business data packet is de-duplicated, the repeated user information is effectively filtered, and the repeated information is avoided to be introduced into the relevant shard, so as to ensure that the cache database and the target database used for shard storage are not caused to waste storage resources and increase storage costs due to repeated information.

[0069] Generally, one piece of user information corresponds to one user, and the user identifier carried by the user information indicates the user dimension. The user identifier carried by the user information, the specified data packet identifier corresponding to the user information, and the indicated data version information are spliced to obtain a feature object corresponding to the user information. In combination with the related description of the “specified data packet identifier” and “data version information” in the foregoing steps S301-S304, the information of the user dimension is not only integrated into the feature object, but also the information of the business dimension and the version dimension is integrated into the feature object. Therefore, the feature object can realize more accurate representation of the user information, and using the feature object as the matching object of the user information de-duplication can also ensure the accuracy and efficiency of de-duplication. Based on the feature object corresponding to the user information, whether there is user information matched with the feature object in the cache database and the target database can be found by using the Bloom filter. When there is matched user information, the user information is removed from the multiple pieces of user information of the to-be-processed business data packet, so as to realize de-duplication. For example, the user information x in the to-be-processed business data packet, the user information x corresponds to the feature object x. It can be understood that the user information x carries the user ID as variable 1, the packet ID as variable 2, and the version number as variable 3, and the feature object x is spliced by using the above three variables. The feature object x is matched with the feature objects corresponding to the user information currently stored in the cache database and the target database. If there is, it means that the cache database or the target database already has the user information x.

[0070] In actual application, referring to Figure 6 , 8 If the execution subject of the shard processing scheme for the business data packet provided in the application is regarded as a number packet system, the system includes a number packet writing service and a de-duplication service. The de-duplication service. The to-be-processed business data packet can be obtained by the number packet writing service, and the number packet writing service is used to judge whether the to-be-processed business data packet needs to be de-duplicated. When it is judged that de-duplication is needed, the de-duplication service is requested to perform de-duplication. For the number packet writing service, if the to-be-processed business data packet indicates the first business data packet of the relevant business or the relevant version, it can be judged that de-duplication is not needed.

[0071] In an exemplary embodiment, the service end can take out a service data packet to be processed from a message queue. The message queue is used to store service data packets carried by received service requests, and each service request indicated by the received service requests has at least two data packet identifiers.

[0072] The message queue here can come from Kafka. By using the message queue to store service data packets carried by received service requests, the peak can be flattened, and the service end can not be affected by the push speed of the upstream service party. The received service request here is not limited to multiple service requests sent in batches by the same upstream service party, but also includes multiple service requests sent in batches by different upstream service parties.

[0073] In actual application, referring to Figure 6 、 8 -10, if the execution subject of the fragmentation processing scheme for service data packets provided in the present application is regarded as a number packet system, the system includes a number packet push service, Kafka, and a number packet write service. The number packet push service receives service requests sent from the upstream service party and sends the service data packets in the service requests to Kafka. The number packet write service as a consumer can obtain service data packets to be processed from Kafka to consume data. In order to adapt to the change of the performance of the write service, the number of number packet write services as consumers can be increased or reduced as needed, and one number packet write service can be regarded as one or a group of machine devices.

[0074] S202: determining a shard identifier corresponding to the plurality of pieces of user information according to the number of the plurality of pieces of user information, the number of the stored pieces of user information, and a preset number threshold; wherein the data packet identifier corresponding to the stored pieces of user information is the specified data packet identifier, and the preset number threshold indicates an upper limit of the number of user information in a shard provided by a target database providing a shard read service;

[0075] In the embodiments of the present application, the server determines the shard identifier corresponding to the plurality of user information according to the number of the plurality of user information, the number of the stored user information, and the preset number threshold, that is, determines the corresponding shard identifier for the to-be-processed service data packet. Different service data packets include different numbers of user information, and the user information indicating the same data packet identifier is integrated for storage by using the shard idea. The preset number threshold indicating the shard can be used to regulate the storage structure of the user information to form a shard that facilitates to improve the read service performance. In the case that the number of the to-be-stored user information, the number of the stored user information, and the upper limit of the number of user information in a shard are known, the shard identifier is allocated for the to-be-stored user information. By using the number of the stored user information and the upper limit of the number of user information in a shard, the identifier of the formed shard and the identifier of the shard still in formation can be known, so as to determine the corresponding shard identifier for the to-be-stored user information in combination with the number of the to-be-stored user information. The shard identifier can be understood as which serial number of shard the to-be-stored user information will be used as a constituent object.

[0076] Compared with the cache database, the target database can be regarded as the underlying storage that provides shard read service. The above-mentioned preset number threshold is limited to the type of the target database, and different types of target databases have different upper limits of the number of user information in a shard. The to-be-processed service data packet comes from a service request received by the server, and when the target database corresponding to the plurality of user information in the to-be-processed service data packet is determined, the following steps can be performed: determining the service identifier corresponding to the service request; determining the target database corresponding to the service identifier based on the preset configuration information; wherein the preset configuration information records the mapping relationship between the preset service identifier and the candidate database. The service identifier here can be the first service identifier appID mentioned in the above steps S301-S304. The mapping relationship recorded by the preset configuration information comprehensively considers the related business conditions of the upstream business party (such as security requirements, real-time requirements) and the related index advantages of the candidate database (such as cost, performance). By using the preset configuration information to determine the target database, the determination efficiency of the target database can be improved, and the determined target database can also meet the related business requirements of the upstream business party. Of course, the preset configuration information can be updated as needed. After the target database is determined, the target database information can be associated with the service data packet. Correspondingly, the target database information indicated by the to-be-processed service data packet can be known, so that the preset number threshold corresponding to the target database is obtained to determine the shard identifier. The preset number threshold corresponding to the target database can be set according to the data amount that the target database can bear without the need for automatic sharding.

[0077] In addition, the index advantage (score interval 1-5, "5" indicates the greatest advantage, and "1" indicates the smallest advantage) corresponding to different types of target databases can refer to the following Table 1:

[0078] Profiling Cost Performance Multiple versions Maintainability Disaster recovery Resource isolation Redis 1 5 3 5 5 5 COS 5 4 5 5 5 2 BDB 2 3 4 5 5 5 Dcache 2 3 3 5 5 3 Weight 0.2 0.3 0.1 0.1 0.1 0.1

[0079] Table 1

[0080] In an exemplary embodiment, as shown in Figure 4 determining the shard identifier corresponding to the plurality of user information according to the number of the plurality of user information, the number of the stored user information, and a preset number threshold value comprises:

[0081] S401: write the number of the plurality of user information into the associated object of the specified data packet identifier in the index database; wherein the index database is used to record the relationship between the data packet identifier and its associated object;

[0082] S402: determine the number of the stored user information indicating the specified data packet identifier in the index database;

[0083] S403: determine the addition result of the number of the stored user information and the number of the plurality of user information by using the index database;

[0084] S404: determine the division result of the addition result and the preset number threshold value by using the index database, and store the division result as the shard identifier corresponding to the plurality of user information.

[0085] As shown in Figure 11 For "the index database is used to record the relationship between the data packet identifier and its associated object", reference can be made to the related description in the foregoing steps S301-S304, and will not be repeated. The number of the plurality of user information (i.e. the number of the user information to be stored) p (p is an integer greater than 0) can be written into the associated object of the specified data packet identifier in the index redis by the server. The server can determine the number q of the stored user information indicating the specified data packet identifier in the index redis, and the number of the stored user information exists as the associated object of the specified data packet identifier. The server determines the addition result p+q of the number of the stored user information and the number of the plurality of user information by using the index redis. The server determines the division result (p+q) / n=m of the addition result and the preset number threshold value by using the index redis, and can store the division result m as the shard identifier corresponding to the plurality of user information in the associated object of the specified data packet identifier, wherein n is an integer greater than 0, and m is an integer greater than or equal to 0. It should be noted that if there is a remainder in the division operation, the division result here indicates the quotient value.

[0086] For example, the shard identifier can be in the form of a serial number, which can be incremented from 0. If the preset quantity threshold (i.e., the shard size) n is 1000, q = 1200, and p = 200, then m = 1400 / 1000, and m is 1. It can be understood that the shard identifier 1 assigned to the plurality of user information currently exists shard identifier 0. The significance of storing the division result as the shard identifier corresponding to the plurality of user information can be to record the number of currently opened shards by using the index redis. Further, q can be updated to 1400. Of course, for the case where the number p of the user information to be stored is greater than the preset quantity threshold n, if p = 1500, then the shard identifier corresponding to any 799 pieces of user information in the user information to be stored is m = (1200 + 799) / 1000, and m is 1, and the shard identifier corresponding to the remaining 701 pieces of user information in the user information to be stored is m = (1200 + 799 + 701) / 1000, and m is 2.

[0087] In addition, the serial number is incremented from 0 is limited to the scope of the same version. When the user information to be stored indicates the package ID: yyyyyy and the version V3.0, if the shard identifier under the current V3.0 has been incremented from 0 to 20, then the shard identifier determined for the user information to be stored is based on the "20". When the user information to be stored indicates the package ID: yyyyyy and the version V4.0, if the shard identifier under the current V3.0 has been incremented from 0 to 20, then the shard identifier determined for the user information to be stored is restarted from the shard identifier "0" under the V4.0. For the version maintenance of the user information, please refer to Figure 12 .

[0088] The index database is introduced to record the related information under the package identifier dimension, which also provides convenience for the query of the related information. Since the number of user information included in different business data packages is different, when the user information indicating the same data package identifier is integrated by using the shard idea, the index database as the provider of the related information ensures the accuracy of determining the shard identifier.

[0089] In actual application, please refer to Figure 6 、 8 -9, 11, if the execution subject of the shard processing scheme for the business data package provided in the present application is regarded as a number package system, the system includes a number package write service and an index redis. The number package write service writes the related information (such as the number of the plurality of user information) of the obtained business data package to be processed into the index redis.

[0090] S203: write the plurality of user information into the cache database according to the shard identifier;

[0091] In the embodiment of the present application, the server writes the plurality of user information into the cache database according to the shard identifier. The cache database is a transit object, and the cache database stores shards that have been formed and shards that are being formed. For example, the plurality of user information corresponds to a shard identifier 2, which indicates that the plurality of user information will be used to form the shard 2. The plurality of user information can independently complete the formation of the shard 2, or can jointly complete the formation of the shard 2 with other user information. The other user information can be user information that has been stored in the cache database, or can be user information that will be stored in the cache database later.

[0092] In actual application, referring to Figure 6 、 8 , 10-11, if the execution subject of the shard processing scheme for the service data packet provided in the present application is regarded as a number packet system, the system includes a number packet writing service and a data cache redis. The number packet writing service writes the plurality of user information of the service data packet to be processed obtained into the data cache redis.

[0093] S204: When the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold, the shard corresponding to the shard identifier is migrated to the target database.

[0094] In the embodiment of the present application, when the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold, the server migrates the shard corresponding to the shard identifier to the target database. After the shard is formed, it can be considered that the shard meets the migration condition, so as to be migrated from the cache database as a transit object to the target database as a final storage object. Referring to Figure 10 , the shard processing scheme for the service data packet provided in the present application realizes the separation of read and write, organizes during writing according to shards, reads one entire shard at a time during reading, and a shard can be regarded as a large string in storage, thereby ensuring the realization of high read speed.

[0095] Considering the difficulty in judging whether the shard is formed, the number of user information in the shard corresponding to the shard identifier meeting the requirement of the preset number threshold can be determined by at least one of the following conditions being met:

[0096] 1) When the user information indicating the adjacent shard identifier starts to be written in the cache database, it is determined that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold; wherein the determination time of the adjacent shard identifier is later than the determination time of the shard identifier.

[0097] Considering the incremental logic of determining the shard identifier, in the shard identifier of 10 and the shard identifier of 11, the determination timing of the latter is later than that of the former. The shard No. 11 exists as the adjacent shard of the shard No. 10 in the cache database. If the user information currently written is used to form the shard No. 11, it can be explained that the shard No. 10 has been formed, the number of user information in the shard No. 10 meets the requirement of the preset number threshold, and the shard No. 10 can be migrated.

[0098] 2) When the difference between the current time and the time of the last write indicating the user information of the shard identifier is greater than the preset time threshold, it is determined that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold;

[0099] The upstream business party sends a plurality of service requests indicating the same service to the server in batches. Generally, the time difference between batches will not be too large, especially when the plurality of service requests indicate the same version. At the same time, the server processes the service data packets carried by the service requests relatively timely. If the difference between the current time t1 and the time t0 of the last write indicating the user information of a shard identifier in the cache database t1-t0 is greater than the preset time threshold T (such as 1 minute, 2 minutes), it can be explained that the shard corresponding to the shard identifier has been formed, the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold, and the shard can be migrated. Or it is explained that the service data packets carried by the plurality of service requests indicating the same service sent by the upstream business party in batches have been stored, or the service data packets carried by the plurality of service requests indicating the same service and the same version sent by the upstream business party in batches have been stored. Even if the number of user information in the shard corresponding to the shard identifier does not meet the requirement of the preset number threshold, the shard can also be migrated.

[0100] 3) When the plurality of user information points to the end flag, it is determined that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold;

[0101] For the service data packets carried by the plurality of service requests respectively sent by the upstream service party in batches to indicate the same service, the sum of the user information can not be an integer multiple of the preset data threshold (i.e., the size of the shard). Therefore, there is a situation that "the service data packets carried by the plurality of service requests respectively sent by the upstream service party in batches to indicate the same service have been completely stored, or the service data packets carried by the plurality of service requests respectively sent by the upstream service party in batches to indicate the same service and the same version have been completely stored, but the number of user information in the shard corresponding to the shard identifier does not meet the requirement of the preset number threshold". When the plurality of user information points to the end flag, it can be indicated that the above-mentioned complete storage has been achieved. Even if the number of user information in the shard corresponding to the shard identifier does not meet the requirement of the preset number threshold, the shard can be migrated.

[0102] By using 1)-3) above, it can be determined in time and effectively whether the shard is formed and whether the migration condition is met, so as to ensure the timeliness of migration. When the cache database uses the memory storage system, considering the high cost of the memory storage system, timely migration can improve the availability of the cache database under the consideration of cost and storage data volume.

[0103] In actual application, the upstream service party sends the plurality of service requests to the server in batches to indicate the same service. For the plurality of service requests indicating the same version, the upstream service party can send an end instruction to the server after sending the service requests to the server. If the execution subject of the shard processing scheme for the service data packet provided in the present application is regarded as a number packet system, the system includes an index redis, a number packet pushing service, a number packet writing service, a Kafka, a checking service and a data cache redis. The end instruction is intended to inform the number packet system that the service party has finished pushing data. Referring to Figure 6 、 8, 10-11, the end instruction carries with the appID (corresponding to the first service identifier described above), thirdID (corresponding to the second service identifier described above), data version, user information total number, etc., and requests the end interface of the number package push service. For example, the upstream service party sends multiple service requests 31-66 indicating the same service and indicating the same version to the server in batches. The user information total number indicates the sum of the user information carried by the service requests 31-66 respectively. The number package push service can write the user information total number into the associated object of the relevant package ID in the index redis, so as to check the number of stored user information. The number package push service can send the end instruction to Kafka, and specifically generate an end flag corresponding to the end instruction, and establish an association between the end flag and the relevant to-be-processed service data package. Correspondingly, when the number package write service consumes data, the end instruction is read and written into the associated object of the relevant package ID in the index redis. The end instruction recorded in the index redis can guide the check service to determine whether the shards of the user information in the data cache redis form and whether the migration condition is met, so as to perform shard migration in time. The shard migration process performed by the check service includes reading the shards that meet the migration condition from the data cache redis, and then calling the write interface of the storage SDK to write the shards to the final storage.

[0104] The target database for the final storage object is not limited to an in-memory storage system; it can support multiple storage types and can adopt other more cost-effective storage systems. This better balances cost and the need to store multi-version data. The data sharding processing scheme for business data packets provided in this application optimizes the storage process and structure to achieve high-speed reading of large amounts of data. Business requests pushed by the business party, carrying data of variable size, are converted into shards of fixed size, allowing for reading an entire shard at a time to improve read performance. The shard sizes are uniform, ensuring stable scheduling speed in scenarios where scheduling is based on the number of shards. The shard size is controllable, adopting different shard sizes based on different underlying storage characteristics to improve read throughput. To adapt to changes in read service performance, the number of machines performing sharded reading can be increased or decreased as needed. Simultaneously, the scheme introduces the in-memory storage system Redis as an intermediate cache (cached database), and the data can ultimately be written to a disk-based storage system, minimizing costs while ensuring high-speed reading. This scheme implements a real-time data sharding process, suitable for scenarios that guarantee high-speed reading of large amounts of data at a relatively low cost. This technology enables high-speed reading of large amounts of data. Current testing (taking the scenario of real-time reading of user information when an upstream business pushes news as an example) shows that it can read 500 million user information records in 1 minute, with a peak speed of 9.12 million user information records per second.

[0105] From the technical solutions provided by the embodiments of the present application, it can be seen that in the embodiments of the present application, the service data packet to be processed is acquired, then the shard identifier corresponding to the plurality of user information is determined according to the quantity of the plurality of user information, the quantity of the stored user information, and a preset quantity threshold, the plurality of user information is written into the cache database according to the shard identifier, and when the quantity of the user information in the shard corresponding to the shard identifier meets the requirement of the preset quantity threshold, the shard corresponding to the shard identifier is migrated to the target database. The service data packet to be processed is the content carried by any service request in the plurality of service requests sent in batches, the plurality of service requests indicate the same data packet identifier, the data packet identifier corresponding to the stored user information is the same as the data packet identifier corresponding to the plurality of user information. The present application is applied to the scene of sending service requests in batches by a service party, wherein each service request carries a service data packet, and one service data packet includes a plurality of user information. The user information in the service data packet is stored in two levels based on the shard idea. In the face of the situation that the quantity of the user information indicated by different service requests is different, the received user information is integrated in a fixed size by using the preset quantity threshold indicating the size of the shard, and the shard meeting the requirement is migrated from the cache to the target database to facilitate the implementation of the shard reading service, so that the reading performance can be improved. On this basis, the target database for final storage can not be limited to using a memory type storage system, can support multiple storage types, and can use other types of storage systems that are more cost-saving.

[0106] The embodiments of the present application also provide a shard processing device for a service data packet, as shown in Figure 13 The shard processing device 1300 for a service data packet includes:

[0107] The acquisition module 1301 is configured to acquire a service data packet to be processed; wherein the service data packet to be processed includes a plurality of user information indicating a specified data packet identifier, the service data packet to be processed is the content carried by any service request in a plurality of service requests sent in batches, and the plurality of service requests all indicate the specified data packet identifier;

[0108] The determination module 1302 is configured to determine a shard identifier corresponding to the plurality of user information according to the quantity of the plurality of user information, the quantity of the stored user information, and a preset quantity threshold; wherein the data packet identifier corresponding to the stored user information is the specified data packet identifier, and the preset quantity threshold indicates the upper limit of the quantity of the user information in the shard provided by a target database for a shard reading service;

[0109] The cache module 1303 is configured to write the plurality of user information into a cache database according to the shard identifier;

[0110] Migrate the shard corresponding to the shard identifier to the target database when the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold.

[0111] It should be noted that the device in the device embodiment and the method embodiment are based on the same inventive concept.

[0112] The electronic device provided in the embodiments of the present application includes a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the shard processing method for service data packets provided in the above method embodiments.

[0113] Further, Figure 14 A hardware structure schematic diagram of an electronic device for implementing the shard processing method for service data packets provided in the embodiments of the present application is shown, and the electronic device can constitute or include the shard processing device for service data packets provided in the embodiments of the present application. As shown in Figure 14 The electronic device 140 can include one or more (in the figure, 1402a, 1402b, …, 1402n are used to show) processors 1402 (the processor 1402 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1404 for storing data, and a transmission device 1406 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. Those skilled in the art can understand that, Figure 14 The structure shown is only a schematic, which does not limit the structure of the above-mentioned electronic device. For example, the electronic device 140 can include more or fewer components than those shown in Figure 14 or have a different configuration from that shown in Figure 14 .

[0114] It should be noted that the one or more processors 1402 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any combination thereof. In addition, the data processing circuit can be a single independent processing module, or any one of the other elements combined into the electronic device 140 (or mobile device) in whole or in part. As referred to in the embodiments of the present application, the data processing circuit serves as a processor to control (for example, selection of a variable resistance terminal path connected to an interface).

[0115] The memory 1404 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the method for processing fragments of service data packets in the embodiments of the present application, and the processor 1402 executes various functional applications and data processing by running the software programs and modules stored in the memory 1404, that is, implements the method for processing fragments of service data packets described above. The memory 1404 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 1404 can further include memories remotely arranged with respect to the processor 1402, which can be connected to the electronic device 140 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0116] The transmission device 1406 is used to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the electronic device 140. In one example, the transmission device 1406 includes a network interface controller (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one embodiment, the transmission device 1406 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0117] The display can be, for example, a touch screen type liquid crystal display (LCD) which can enable a user to interact with a user interface of the electronic device 140 (or a mobile device).

[0118] The embodiments of the present application also provide a computer readable storage medium which can be arranged in an electronic device to save at least one instruction or at least one program for implementing a method for processing fragments of service data packets in the method embodiments, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the method for processing fragments of service data packets provided by the method embodiments.

[0119] Optionally, in the present embodiment, the storage medium described above can be located in at least one of a plurality of network servers of a computer network. Optionally, in the present embodiment, the storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media which can store program codes.

[0120] It should be noted that the above-mentioned order of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multi-task processing and parallel processing are possible or can be advantageous.

[0121] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, for the device and electronic equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0122] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0123] The above-mentioned is only the preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing fragmentation of service data packets, characterized by, The method comprises: obtaining a to-be-processed service data packet, the to-be-processed service data packet comprising a plurality of pieces of user information indicating a specified data packet identifier, the to-be-processed service data packet being content carried by any one of a plurality of service requests sent in batches, the plurality of service requests all indicating the specified data packet identifier; writing a quantity of the plurality of pieces of user information into an associated object of the specified data packet identifier in an index database, the index database being used to record a relationship between a data packet identifier and an associated object thereof; determining, in the index database, a quantity of stored user information indicating the specified data packet identifier, the stored user information corresponding to a data packet identifier being the specified data packet identifier; determining, by using the index database, a sum of the quantity of the stored user information and the quantity of the plurality of pieces of user information; determining, by using the index database, a quotient of the sum and a preset quantity threshold, the preset quantity threshold indicating an upper limit of a quantity of user information in a shard for which a target database provides shard reading service; and storing the quotient as a shard identifier corresponding to the plurality of pieces of user information. writing the plurality of pieces of user information into a cache database according to the shard identifier; when a quantity of user information in a shard corresponding to the shard identifier meets a requirement of the preset quantity threshold, migrating the shard corresponding to the shard identifier to the target database.

2. The method of claim 1, wherein, Before the obtaining of the to-be-processed service data packet, the method further comprises: in response to the received service request, determining a first service identifier, a second service identifier, and data version information corresponding to the service request; in the index database, searching for whether there is a matching data packet identifier according to the first service identifier and the second service identifier, wherein the index database is used to record a relationship between a data packet identifier and an associated object thereof; when there is, taking the matching data packet identifier as the specified data packet identifier, and writing the data version information into a first target associated object, wherein the first target associated object is an associated object of the matching data packet identifier in the index database; when there is not, creating a corresponding data packet identifier based on the first service identifier and the second service identifier to serve as the specified data packet identifier, and writing the first service identifier, the second service identifier, and the data version information into a second target associated object, wherein the second target associated object is an associated object of the corresponding data packet identifier in the index database.

3. The method of claim 1, wherein, The method further comprises: in response to the received service request, determining a service identifier corresponding to the service request; determining, based on preset configuration information, the target database corresponding to the service identifier, wherein the preset configuration information records a mapping relationship between a preset service identifier and a candidate database.

4. The method of claim 1, wherein, After the obtaining of the to-be-processed service data packet, the method further comprises: splicing a user identifier carried by the user information, the specified data packet identifier corresponding to the user information, and the indicated data version information to obtain a feature object corresponding to the user information; based on the feature object corresponding to the user information, determining whether there is user information matching the feature object in the cache database and the target database; when there is, removing the user information from the plurality of user information.

5. The method of claim 1, wherein, The method further comprises: when starting to write user information indicating an adjacent shard identifier in the cache database, determining that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold; or, when the difference between the current time and the time of the last time of writing user information indicating the shard identifier is greater than a preset time threshold, determining that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold; or, when the plurality of user information points to an end flag, determining that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold; wherein the determination time of the adjacent shard identifier is later than the determination time of the shard identifier.

6. The method of claim 1, wherein, The obtained to-be-processed service data packet comprises: taking the to-be-processed service data packet from a message queue; wherein the message queue is used to store service data packets carried by received service requests, and data packet identifiers indicated by each service request in the received service requests have at least two.

7. An apparatus for processing fragmentation of service data packets, characterized by, The device comprises: an acquisition module configured to acquire a to-be-processed service data packet; wherein the to-be-processed service data packet comprises a plurality of user information indicating a specified data packet identifier, and the to-be-processed service data packet is carried by any service request in a plurality of service requests sent in batches, and the plurality of service requests all indicate the specified data packet identifier; a determination module configured to write the number of the plurality of user information into an associated object of the specified data packet identifier in an index database, the index database is used to record the relationship between data packet identifiers and their associated objects; determine the number of stored user information indicating the specified data packet identifier in the index database, the stored user information corresponding to a data packet identifier is the specified data packet identifier; determine the addition result of the number of the stored user information and the number of the plurality of user information by using the index database; determine the division result of the addition result and a preset number threshold by using the index database, and store the division result as a shard identifier corresponding to the plurality of user information, the preset number threshold indicates the upper limit of the number of user information in a shard provided by a target database for shard reading service; a cache module configured to write the plurality of user information into a cache database according to the shard identifier; a migration module configured to migrate the shard corresponding to the shard identifier to the target database when the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold.

8. The apparatus of claim 7, wherein, Before acquiring the to-be-processed service data packet, the device is further configured to: determine a first service identifier, a second service identifier and data version information corresponding to the service request in response to the received service request; According to the first service identifier and the second service identifier, it is determined whether there is a matched data packet identifier in an index database; wherein the index database is used to record the relationship between a data packet identifier and its associated object; When there is, the matched data packet identifier is taken as the specified data packet identifier, and the data version information is written into a first target associated object; wherein the first target associated object is the associated object of the matched data packet identifier in the index database; When there is not, a corresponding data packet identifier is created based on the first service identifier and the second service identifier to be the specified data packet identifier, and the first service identifier, the second service identifier and the data version information are written into a second target associated object; wherein the second target associated object is the associated object of the corresponding data packet identifier in the index database.

9. The apparatus of claim 7, wherein, The device is also used to: determine the service identifier corresponding to the service request in response to the received service request; determine the target database corresponding to the service identifier based on preset configuration information; wherein the preset configuration information records the mapping relationship between a preset service identifier and a candidate database.

10. The apparatus of claim 7, wherein, After the device acquires the service data packet to be processed, the device is also used to: splice the user identifier carried by the user information, the specified data packet identifier corresponding thereto and the indicated data version information to obtain the feature object corresponding to the user information; determine whether there is user information with a matching feature object in the cache database and the target database based on the feature object corresponding to the user information; when there is, remove the user information from the plurality of user information.

11. The apparatus of claim 7, wherein, The device is also used to: when the user information indicating the adjacent shard identifier starts to be written in the cache database, it is determined that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold; or, when the difference between the current time and the time when the user information indicating the shard identifier was last written is greater than a preset time threshold, it is determined that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold; or, when the plurality of user information points to an end flag, it is determined that the number of user information in the shard corresponding to the shard identifier meets the requirement of the preset number threshold; wherein the determination time of the adjacent shard identifier is later than the determination time of the shard identifier.

12. The apparatus of claim 7, wherein, The acquisition module is also used to: take the service data packet to be processed from a message queue; wherein the message queue is used to store the service data packet carried by the received service request, and the data packet identifier indicated by each service request in the received service request has at least two.

13. An electronic device, the electronic device comprising a processor and a memory, wherein, The memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the shard processing method for service data packets according to any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the fragmentation processing method for service data packets as claimed in any one of claims 1-6.

15. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer readable storage medium, and the processor of the computer device reads and executes the computer instructions from the computer readable storage medium, so that the computer device executes the fragmentation processing method for service data packets as claimed in any one of claims 1-6.

Images