Data processing method, device and equipment and readable storage medium

Abstract

The invention discloses a data processing method and device, equipment and a readable storage medium, and the method comprises the steps: obtaining a deduplication processing request triggered by a target business service, the deduplication processing request at least carrying a receiving object identifier, a deduplication time range, deduplication information and a target business service identifier; determining partition creation parameters of the distributed bloom filter for the target business service from pre-cached target de-reconfiguration information; determining a corresponding routing partition identifier according to the receiving object identifier and the partition creation parameter, and selecting at least one target partition corresponding to the deduplication time range in the distributed service system according to the target business service identifier and the routing partition identifier; and sending the receiving object identifier and the de-duplication information to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition executes de-duplication detection based on the receiving object identifier and the de-duplication information. Therefore, the deduplication of each business service can be managed in a unified manner, and the deduplication processing efficiency and accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a data processing method, apparatus, device, and readable storage medium. Background Technology

[0002] The development of information technology has driven the growth of online business service industries, such as online shopping, e-ticketing, and gaming. Most online business service industries involve information push services. However, in any business service's information push service, attention must be paid to the problem of duplicate information pushes. To avoid duplicate information pushes within a short period, deduplication processing needs to be performed before pushing the information.

[0003] Related technologies prevent the insertion of duplicate data by setting uniqueness constraints at the database level, or by storing elements using hash tables and determining whether the stored elements are duplicates based on time complexity, or by using data structures from third-party services for deduplication.

[0004] In the process of researching and practicing related technologies, the inventors of this application found that the related technologies perform deduplication detection through database constraints, hash tables or third-party services, which are only applicable to some business services with basic deduplication detection needs and cannot meet the deduplication detection needs of a wide range of business services. Furthermore, due to the inconsistency of deduplication detection standards between different business services, it is difficult to uniformly manage the deduplication detection of each business service, thereby reducing the efficiency and accuracy of deduplication detection for each business service. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and readable storage medium that can meet the deduplication needs of a wide range of business services, realize unified management of deduplication for various business services, and improve the efficiency and accuracy of deduplication for various business services.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0007] This application provides a data processing method, including:

[0008] Obtain the deduplication request triggered by the target business service. The deduplication request carries at least the receiving object identifier, the deduplication time range, the deduplication information, and the target business service identifier.

[0009] Based on the target service identifier, determine the partition creation parameters of the distributed Bloom filter for the target service from the pre-cached target deduplication configuration information;

[0010] The corresponding routing partition identifier is determined based on the receiving object identifier and the partition creation parameters, and at least one target partition corresponding to the deduplication time range is selected in the distributed service system based on the target business service identifier and the routing partition identifier.

[0011] The distributed service system includes pre-created partitions for deduplication configuration information of different business services, and a Bloom filter associated with each partition.

[0012] The received object identifier and the deduplication information are sent to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the received object identifier and the deduplication information.

[0013] Accordingly, embodiments of this application provide a data processing apparatus, including:

[0014] The acquisition unit is used to acquire a deduplication request triggered by the target business service. The deduplication request carries at least the receiving object identifier, the deduplication time range, the deduplication information, and the target business service identifier.

[0015] The determining unit is configured to determine the partition creation parameters of the distributed Bloom filter for the target business service from the pre-cached target deduplication configuration information based on the target business service identifier.

[0016] The selection unit is used to determine the corresponding routing partition identifier based on the receiving object identifier and the partition creation parameters, and to select at least one target partition corresponding to the deduplication time range in the distributed service system based on the target service identifier and the routing partition identifier.

[0017] The distributed service system includes pre-created partitions for deduplication configuration information of different business services, and a Bloom filter associated with each partition.

[0018] The sending unit is configured to send the receiving object identifier and the deduplication information to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the receiving object identifier and the deduplication information.

[0019] In some embodiments, the selected unit is further configured to:

[0020] Based on the target business service identifier, determine the partition pre-created for the target business service in the distributed service system;

[0021] Query multiple candidate partitions corresponding to the routing partition identifier from the pre-created partitions;

[0022] From the plurality of candidate partitions, determine at least one target partition corresponding to at least one target deduplication time period included in the deduplication time range.

[0023] In some embodiments, the selected unit is further configured to:

[0024] Determine multiple target deduplication time periods corresponding to the deduplication time range;

[0025] From the multiple candidate partitions, select multiple target partitions corresponding to the multiple target deduplication time periods.

[0026] In some embodiments, the selected unit is further configured to:

[0027] Based on the partition creation parameters, determine the number of partitions created by the distributed Bloom filter of the target business service per unit time period;

[0028] A hash calculation is performed on the identifier of the receiving object to obtain a candidate hash value;

[0029] The candidate hash value is moduloed according to the number of partitions created to obtain the routing partition identifier.

[0030] In some embodiments, the determining unit is further configured to:

[0031] By performing a deduplication check service process, the pre-cached target deduplication configuration information corresponding to the target service identifier is retrieved from the cache space;

[0032] Based on the target deduplication configuration information, determine the partition creation parameters when creating a distributed Bloom filter for the target business service.

[0033] In some embodiments, the data processing apparatus further includes a loading unit for:

[0034] The deduplication check service process loads the target deduplication configuration information corresponding to the target business service identifier from the deduplication configuration information public terminal.

[0035] The deduplication configuration information public terminal includes at least one business service for deduplication configuration information input for deduplication checks;

[0036] The loaded target deduplication configuration information is cached in the cache space.

[0037] In some embodiments, the data processing apparatus further includes a creation unit, which is also used for:

[0038] Load the target deduplication configuration information corresponding to the target service identifier from the public terminal of the deduplication configuration information;

[0039] Based on the target deduplication configuration information, determine the creation address, deduplication duration, and partition creation parameters of the distributed Bloom filter;

[0040] Based on the target business service identifier, the deduplication duration information, and the partition creation parameters, a Bloom filter corresponding to the partition is created in the creation address of the distributed service system.

[0041] In some implementations, the creation unit is further configured to:

[0042] Based on the target business service identifier, the deduplication duration information, and the partition creation parameters, generate multiple filter indexes corresponding to at least one deduplication time period;

[0043] For each deduplication time period, multiple filter indexes are sequentially detected in the creation address of the distributed service system to obtain the detection results.

[0044] Based on the detection results, a target filter index that has not been created with a corresponding Bloom filter is identified, and a Bloom filter for the partition corresponding to the target filter index is created in the creation address of the distributed service system.

[0045] In some implementations, the creation unit is further configured to:

[0046] The deduplication time range is determined based on the deduplication duration information;

[0047] Based on the partition creation parameters, determine multiple partition identifiers to be created corresponding to a deduplication time period;

[0048] For each deduplication period within the deduplication period range, multiple filter indexes are constructed based on the target business service identifier and the identifier of each partition to be created.

[0049] In some embodiments, the acquiring unit is further configured to:

[0050] Obtain the deduplication processing request input to the target interface of the target business service by deduplicating the service process;

[0051] The request parameters defined by the target interface include at least the receiving object identifier, deduplication information, deduplication time range, and target business service identifier.

[0052] In some implementations, the target interface includes a deduplication write interface, and the data processing device further includes a push unit, which is also used for:

[0053] When the deduplication check results of the Bloom filter corresponding to each target partition are all received and do not contain deduplication information index, determine the target Bloom filter of the target partition that matches the current time information.

[0054] Construct a deduplication information index based on the received object identifier and the deduplication information;

[0055] The deduplication information index is sent to the target Bloom filter for storage, and the target content corresponding to the deduplication information is pushed to the target receiving object corresponding to the receiving object identifier.

[0056] Furthermore, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data processing method.

[0057] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the aforementioned data processing method.

[0058] Furthermore, this application also provides a computer program product, which includes a computer program that can be stored in a storage medium. A processor of a computer device reads the computer program from the storage medium and executes the computer program to implement the aforementioned data processing method.

[0059] This application embodiment can obtain a deduplication request triggered by a target business service. The deduplication request carries at least a receiving object identifier, a deduplication time range, deduplication information, and a target business service identifier. Based on the target business service identifier, the partition creation parameters of the distributed Bloom filter for the target business service are determined from the pre-cached target deduplication configuration information. Based on the receiving object identifier and the partition creation parameters, the corresponding routing partition identifier is determined, and based on the target business service identifier and the routing partition identifier, at least one target partition corresponding to the deduplication time range is selected in the distributed service system. The distributed service system includes partitions pre-created for deduplication configuration information of different business services, and a Bloom filter associated with each partition. The receiving object identifier and deduplication information are sent to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the receiving object identifier and the deduplication information.

[0060] Based on this, this application can obtain a deduplication request triggered by any target business service. This deduplication request contains various parameters defined according to a unified interface. Then, it reads the partition creation parameters of the target business service from the cache. These partition creation parameters can at least represent the number of partitions of the distributed Bloom filter created for the target business service within a deduplication time unit. Thus, a routing partition identifier is generated by combining the receiving object identifier. Then, using the routing partition identifier and the deduplication time range as a filtering index, the target partition is selected in the distributed service system. Each target partition corresponds to the routing partition identifier of a deduplication period within the deduplication time range. The distributed service system has multiple partitions and a corresponding Bloom filter for each target business service pre-created. Finally, by sending the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition, the application requests the Bloom filter corresponding to the target partition to perform deduplication checks.

[0061] Therefore, compared to existing technologies that use database constraints, hash tables, or third-party services to perform deduplication checks, which cannot meet the deduplication needs of a wide range of business services, this application pre-creates corresponding distributed Bloom filters based on partition creation parameters for each business service. Each Bloom filter corresponds to a partition, and each partition is divided according to the deduplication period and partition identifier to meet the deduplication detection needs of different business services. Furthermore, each business service can carry various parameters defined by a unified interface in its request to request the Bloom filter of the target partition corresponding to each business service to perform deduplication detection. In this way, unified management and concurrent processing of deduplication processing for each business service are achieved, improving the efficiency and accuracy of deduplication processing for each business service.

[0062] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of this application.

[0065] Figure 2 This is a flowchart illustrating the steps of the data processing method provided in the embodiments of this application;

[0066] Figure 3 This is another step of the data processing method provided in the embodiments of this application;

[0067] Figure 4 Example diagram of the framework for the deduplication check service provided in the embodiments of this application;

[0068] Figure 5 This is a structural example diagram of the deduplication check service system provided in the embodiments of this application;

[0069] Figure 6 This is an example diagram illustrating a scenario where the deduplication SDK loads deduplication configuration information, as provided in this application embodiment.

[0070] Figure 7 This is an example diagram of the page for entering deduplication configuration information provided in an embodiment of this application;

[0071] Figure 8 A flowchart illustrating the creation process of a Bloom filter provided in an embodiment of this application;

[0072] Figure 9 This is an example diagram of the deduplication storage architecture provided in the embodiments of this application;

[0073] Figure 10 This is an example diagram of a partitioned routing scenario for a Bloom filter provided in an embodiment of this application;

[0074] Figure 11 This is a flowchart illustrating the deduplication SDK provided in the embodiments of this application.

[0075] Figure 12 Example diagram of indicator monitoring for the deduplication check service provided in the embodiments of this application;

[0076] Figure 13 Example diagram of capacity monitoring information for a Bloom filter provided in an embodiment of this application;

[0077] Figure 14 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0078] Figure 15 This is a schematic diagram of the terminal structure provided in the embodiments of this application;

[0079] Figure 16 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0080] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0081] It is understood that in the specific implementation of this application, data related to receiving object identifiers, deduplication time ranges, and deduplication information are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the receiving object is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards.

[0082] Furthermore, when this application embodiment needs to obtain data such as the receiving object identifier, deduplication time range, and deduplication information, it will obtain separate permission or separate consent for the receiving object identifier, deduplication time range, and deduplication information through pop-up windows or redirection to a confirmation page. After clearly obtaining separate permission or separate consent for the receiving object identifier, deduplication time range, and deduplication information, it will then obtain the necessary receiving object identifier, deduplication time range, and deduplication information for the normal operation of this application embodiment.

[0083] It should be noted that while some processes described in the specification, claims, and accompanying drawings contain multiple steps that appear in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any particular order of execution. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0084] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0085] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0086] Deduplication: Deduplication is a common step in data processing that refers to removing duplicate records or elements from a dataset so that each record or element appears only once in the dataset.

[0087] Multi-tenancy: Multi-tenancy is a software architecture pattern that allows multiple independent users (called tenants) to share resources on the same application or system while maintaining the isolation of their respective data and configurations.

[0088] SDK: An SDK (Software Development Kit) is a set of tools, libraries, and documentation used to help developers create applications for a specific platform or service.

[0089] Bloom Filter: A Bloom filter is a space- and time-efficient probabilistic data structure used to determine whether an element is in a set. It consists of a hash function and a bit array.

[0090] This application provides a data processing method, apparatus, device, and readable storage medium. Specifically, the data processing methods of various embodiments of this application can be executed by a computer device, which can be a server or a terminal, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a mobile phone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart home appliance, vehicle terminal, smart voice interaction device, aircraft, etc., but is not limited to these.

[0091] This application provides a data processing method that can obtain a deduplication request triggered by any target business service. This deduplication request includes various parameters defined according to a unified interface. Then, it reads the partition creation parameters of the target business service from the cache. These partition creation parameters at least represent the number of partitions of the distributed Bloom filter created for the target business service within a deduplication time period. A routing partition identifier is then generated by combining the receiving object identifier with the deduplication time range. Next, using the routing partition identifier and the deduplication time range as a filtering index, a target partition is selected in the distributed service system. Each target partition corresponds to a routing partition identifier within a deduplication time period in the deduplication time range. The distributed service system pre-creates multiple partitions and corresponding Bloom filters for each target business service. Finally, by sending the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition, the method requests the Bloom filter corresponding to the target partition to perform a deduplication check. This is specifically illustrated in the following embodiments:

[0092] Related technologies prevent duplicate data insertion by setting uniqueness constraints at the database level, storing elements using hash tables and determining duplicates based on time complexity, or using third-party data structures for deduplication. However, these methods are only suitable for some basic deduplication needs of business services and cannot meet the deduplication needs of a wide range of business services. Furthermore, due to inconsistent deduplication standards among different business services, it is difficult to manage deduplication across different services uniformly, thus reducing the efficiency and accuracy of deduplication across various business services.

[0093] To address the aforementioned issues, this application proposes a data processing method. This method pre-creates distributed Bloom filters based on partition creation parameters for each business service. Each Bloom filter corresponds to a partition, and each partition is divided according to a deduplication time period and partition identifier to meet the deduplication detection needs of different business services. Furthermore, each business service can include parameters defined by a unified interface in its request to request the Bloom filters of its corresponding target partition to perform deduplication detection. This achieves unified management and concurrent processing of deduplication for each business service, improving the efficiency and accuracy of deduplication for each service. Please refer to the following specific embodiments for details.

[0094] This application provides a data processing system. The devices in this system may include a server and / or a terminal. The terminal may request the server to execute the data processing method of this application.

[0095] For example, the system includes a server or terminal. The terminal or server can obtain a deduplication request triggered by a target business service. The deduplication request carries at least a receiving object identifier, a deduplication time range, deduplication information, and a target business service identifier. Based on the target business service identifier, the system determines the partition creation parameters of the distributed Bloom filter for the target business service from the pre-cached target deduplication configuration information. Based on the receiving object identifier and the partition creation parameters, the system determines the corresponding routing partition identifier and selects at least one target partition corresponding to the deduplication time range in the distributed service system based on the target business service identifier and the routing partition identifier. The distributed service system includes partitions pre-created for deduplication configuration information of different business services, and a Bloom filter associated with each partition. The system sends the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the receiving object identifier and the deduplication information.

[0096] For example, see Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of this application. The system includes a terminal 110 and a server 120.

[0097] The terminal 110 can send data processing requests to the server 120. In addition, a client can be installed on the terminal 110, and related business services can be enjoyed through the client. This process involves data interaction, such as the client sending an information push request to the server, or the server sending a personalized information recall to the client.

[0098] The server 120 can be a distributed system composed of multiple physical machines, each of which can correspond to a service node or server. For example, the server 120 may include a business server, a deduplication configuration information public terminal, and a management terminal, and / or a distributed service system for creating a distributed Bloom filter, i.e., a Redis service cluster. After receiving a data processing request, server 120 can execute the steps of the data processing method, including obtaining a deduplication request triggered by the target business service. This deduplication request carries at least the receiving object identifier, deduplication time range, deduplication information, and target business service identifier. Based on the target business service identifier, server 120 determines the partition creation parameters of the distributed Bloom filter for the target business service from the pre-cached target deduplication configuration information. Based on the receiving object identifier and the partition creation parameters, server 120 determines the corresponding routing partition identifier and selects at least one target partition corresponding to the deduplication time range in the distributed service system. The distributed service system includes partitions pre-created for different business services based on deduplication configuration information, and a Bloom filter associated with each partition. Server 120 sends the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition, enabling the Bloom filter associated with each target partition to perform deduplication detection based on the receiving object identifier and the deduplication information.

[0099] Based on the above, this application can pre-create corresponding distributed Bloom filters according to the partition creation parameters of each business service. Each Bloom filter corresponds to a partition, and each partition is divided according to the deduplication period and partition identifier to meet the deduplication detection requirements of different business services. Furthermore, each business service can carry various parameters defined by a unified interface in the request to request the Bloom filter of the target partition corresponding to each business service to perform deduplication detection. In this way, the deduplication processing of each business service can be uniformly managed and concurrently processed, improving the deduplication processing efficiency and accuracy of each business service.

[0100] It should be noted that the above are just examples and can be applied to other data processing scenarios, which will not be elaborated here.

[0101] For ease of understanding, each step of the data processing method will be described in detail below. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0102] See Figure 2 , Figure 2 This is a flowchart illustrating the steps of a data processing method provided in an embodiment of this application. In this embodiment, the data processing method is described using an example where it is executed by a server. The specific process is as follows:

[0103] 101. Deduplication requests triggered by obtaining the target business service.

[0104] Many business services involve information push services, such as online shopping, e-ticketing, and gaming. To promote these services, information about the services needs to be pushed to users. However, to improve the quality of information push services, it is necessary to pay attention to the problem of "duplicate pushes." Deduplication should be implemented during the push process to avoid sending duplicate business content to the same user, thereby improving the quality of information push services.

[0105] There are various deduplication methods in related technologies. For example, deduplication can be performed using database constraints. Setting unique constraints or primary keys at the database level can prevent the insertion of duplicate data, but it requires adding additional indexes to the database and specifying unique constraints in advance. This is suitable for more customized scenarios, but its concurrency capabilities are insufficient and it cannot support massive calls. Another example is using hash tables (such as Java's HashSet and Python's Set) in memory to store processed elements. It can determine whether an element is duplicated in O(1) time complexity. However, this method may consume a lot of memory and is limited by the memory of a single machine, making it unsuitable for large-scale data. Yet another example is deduplication using data types provided by third-party services, generally using Redis's Set or Sorted Set structures. This method relies on data structure implementation, and when the data volume is extremely large, there are problems with complex expansion and high storage costs.

[0106] Furthermore, the technologies mentioned above are only applicable to some basic deduplication detection needs of business services, and cannot meet the complex deduplication detection needs of a wide range of business services. Moreover, due to the inconsistency of deduplication detection standards between different business services, it is difficult to manage the deduplication detection of each business service in a unified manner, which reduces the efficiency and accuracy of deduplication detection for each business service.

[0107] Based on this, in order to uniformly manage the deduplication checks of various business services, this application embodiment pre-creates corresponding distributed Bloom filters according to the deduplication check requirements of each business service, and provides a deduplication check function toolkit (SDK). This allows each business service to integrate the deduplication check function through the toolkit. The toolkit provides relevant interfaces, which define multiple parameter types. Each business service only needs to call these interfaces to trigger the deduplication check and provide relevant parameters according to the parameter types defined in the toolkit interfaces. Based on this, the computer device can index the corresponding Bloom filter of the business service based on the parameters provided by the business service, so as to perform deduplication detection in the Bloom filter.

[0108] It should be noted that, in order to uniformly manage the deduplication requirements of various business services and simplify the deduplication process of each business service, this application embodiment provides a toolkit for deduplication checking. Each business service integrates deduplication checking functionality through this toolkit. Based on the parameter types defined in the toolkit's interface, when any business service needs to perform deduplication checking, the relevant parameters of that business service are collected and obtained according to the parameter types defined in the interface for use in the deduplication checking implementation process. In this way, the operators of each business service do not need to understand the storage and operation logic of the deduplication process, lowering the barrier to entry for business service operators to use the deduplication checking function. It has high inclusiveness, meeting the needs of business services with diverse deduplication requirements, and is conducive to the unified management of deduplication processing requirements of business services.

[0109] The deduplication request must include at least the recipient identifier, the deduplication time range, the deduplication information, and the target business service identifier.

[0110] The recipient identifier can be the identifier of the information recipient in the information push process, used to indicate the identity of the information recipient. Specifically, it can be contact information, email address, social media account, name, etc., without limitation here.

[0111] Among them, the deduplication information can be the information content that the corresponding business service needs to deduplicate and filter in the information push. It can be understood as the candidate content to be pushed. Specifically, before the information push is executed, this content needs to be deduplicated and filtered.

[0112] The deduplication time range can be the timeframe during which the corresponding business service performs deduplication on a specific information recipient and the deduplicated information, representing the time length factor to be considered during the deduplication process. Specifically, when performing deduplication, the main focus is on confirming whether the relevant content information (i.e., "deduplicated information") has been pushed or sent to the information recipient. This deduplication process can have time characteristics, such as whether the content of "deduplicated information" has been repeatedly pushed to the information recipient within a certain number of hours, or whether the content of "deduplicated information" has been repeatedly pushed to the information recipient within a certain number of days. Other timeframes are also possible; the above are just examples and are not limited here.

[0113] The target business service identifier can be the identifier of the target business service that triggered the deduplication request. This identifier can be a service name, number, code, address information, contact information, unique identifier, etc., or it can be a hash value calculated based on unique information to represent the identity of the business service and to distinguish this business service from other business services.

[0114] In some implementations, the business service integrates a deduplication check function. The business service provides request parameters according to the target interface of the deduplication check function to generate a deduplication processing request, which is then triggered through the interface of the deduplication check function. For example, step 101 may include: obtaining the deduplication processing request input by the target business service through the target interface of the deduplication check service process; wherein the request parameters defined by the target interface include at least the receiving object identifier, deduplication information, deduplication time range, and target business service identifier.

[0115] The computer device in this application embodiment can be a business server, i.e., a server running business services. These business services on the business server integrate deduplication checking functionality through a deduplication checking toolkit (SDK). Based on this, taking a target business service as an example, when the target business service needs to perform deduplication processing during operation, such as during information push, it calls the target interface of the deduplication checking function. It provides relevant request parameters according to the parameter types defined in the target interface, such as the receiving object identifier, deduplication information, deduplication time range, target business service identifier, etc. Based on these parameters, a deduplication processing request is generated, triggering the deduplication processing process. In this way, unified management of deduplication requirements for multiple business services is achieved. Through the interface method of this SDK, business services only need to provide the corresponding parameter interfaces according to the parameters defined in the interface, thus simplifying the deduplication processing operation.

[0116] By providing a deduplication toolkit, each business service can integrate deduplication functionality. Based on the parameter types defined in the toolkit's interface, when any business service requires deduplication, relevant parameters are collected and retrieved for the deduplication process. This allows for unified management of deduplication requirements across various business services.

[0117] 102. Based on the target business service identifier, determine the partition creation parameters of the distributed Bloom filter for the target business service from the pre-cached target deduplication configuration information.

[0118] In this embodiment, after detecting a deduplication request triggered by the deduplication check function, the configuration information of the deduplication service for the corresponding target business service can be read from the cache space according to the request parameters carried in the deduplication request. Specifically, based on the target business service identifier in the request parameters, the partition creation parameters of the distributed Bloom filter created for the deduplication configuration requirements of the target business service can be obtained from the cache space, so as to participate in the routing calculation of the partition of the Bloom filter and determine the execution partition of the deduplication request.

[0119] It should be noted that, in order to address the deduplication check requirements of different business services, a deduplication configuration information input system, namely a deduplication configuration information public terminal, is provided for multiple business services. This system provides deduplication configuration input services for each business service, allowing operators of each business service to input their respective deduplication configuration information on the corresponding deduplication configuration input page of the deduplication configuration information public terminal. This deduplication configuration information is then stored on the deduplication configuration information public terminal, enabling the reporting of each deduplication request to the deduplication configuration information public terminal. On the one hand, the relevant backend reads the deduplication configuration information to create a distributed Bloom filter. On the other hand, it pulls the deduplication configuration information for caching, and calculates the target Bloom filter to which subsequent deduplication check requests should be assigned based on the deduplication configuration information in the cache, i.e., the Bloom filter address of the route, such as the identifier of the partition corresponding to the Bloom filter.

[0120] It should be noted that the server in this application embodiment can be a distributed system type server, including multiple sub-servers, such as a deduplication configuration information public terminal for storing deduplication configuration information entered by each business service, a business server for running business services, a service cluster of Bloom filters corresponding to business services, a management server for managing and creating Bloom filters for business services, etc. All of the above can be understood as computer devices in this application embodiment.

[0121] The target deduplication configuration information can be the deduplication configuration information corresponding to the target business service. This configuration information is entered by the operator of the target business service into the public deduplication configuration information terminal, reflecting the deduplication requirements of the target business service. For example, this target deduplication configuration information may include partition creation parameters, peak deduplication read / write speed (the number of Bloom filters determines the read / write speed), deduplication storage service address (which can be understood as the creation address of the Bloom filters), maximum deduplication time, and other fields. The above fields of the target deduplication configuration reflect and represent the deduplication requirements of the target business service. The above is only an example and is not intended as a specific limitation.

[0122] The partition creation parameter can be understood as the number of partitions created for the distributed Bloom filter targeting the target business service within a unit of deduplication time period. Specifically, the Bloom filter service cluster (Redis cluster) contains multiple Bloom filters corresponding to each business service in each deduplication time period. That is, the Bloom filters for business services have time-related characteristics and are distinguished according to these characteristics. Specifically, for each business service, the deduplication configuration information stipulates that a separate Bloom filter for each deduplication time period must be created; that is, the Bloom filters are updated according to the unit of deduplication time period.

[0123] For example, if the deduplication configuration information of a business service requires the creation of 10,000 Bloom filters for a deduplication period of one day, then 10,000 partitions need to be created for the business service every day, and each partition corresponds to one Bloom filter. The partition creation parameter can be 10,000 Bloom filters / day.

[0124] In this embodiment, to meet the different deduplication requirements of multiple service providers, the operators of each service provider input their deduplication configuration information on the deduplication configuration information input page corresponding to the deduplication configuration information public terminal, so as to store the deduplication configuration information on the public terminal. At this time, it is necessary to synchronize and cache the deduplication configuration information of each service provider so that when a service provider has a deduplication requirement, it can generate a deduplication check request based on the cached deduplication configuration information and send the deduplication check request to the determined Bloom filter.

[0125] In some implementations, the target deduplication configuration information entered by the target business service can be retrieved periodically from the deduplication configuration information public terminal to cache the target deduplication configuration information. For example, before step 102, it may include: loading the target deduplication configuration information corresponding to the target business service identifier from the deduplication configuration information public terminal through the deduplication check service process; wherein, the deduplication configuration information public terminal contains deduplication configuration information for at least one business service input for deduplication check; and caching the loaded target deduplication configuration information to the cache space.

[0126] Specifically, the target business service integrates deduplication checking functionality through a deduplication checking toolkit (SDK). This function can run as a service process dependent on the target business server. The deduplication checking service process periodically retrieves target deduplication configuration information from a public deduplication configuration information repository and caches this information in the business server's cache space. Additionally, the deduplication checking toolkit can have its own internal cache space, allowing the target deduplication configuration information to be cached there as well. This facilitates the rapid retrieval of the target deduplication configuration information when the target business service requires deduplication, allowing for the retrieval of relevant parameters, such as partition creation parameters, and enabling fast execution of deduplication checks based on the cached target deduplication configuration information.

[0127] In this embodiment, to meet the high-concurrency deduplication query requirements of business services, multiple distributed Bloom filters can be created separately for each business service according to the corresponding deduplication configuration information. In this way, the storage of deduplicated data is achieved by creating multiple distributed Bloom filters to cope with the high-concurrency deduplication check requirements and scalability. Subsequently, when faced with a large number of business objects (users of the business service) of the target business service, the data of these business objects can be partitioned and managed, such as distributing the data of these business objects evenly to multiple distributed Bloom filters. That is, each Bloom filter stores and manages the data of a specific group of business objects. For example, the first Bloom filter stores and manages the data of business object 1, business object 2, and business object 3, and the second Bloom filter stores and manages the data of business object 4, business object 5, and business object 6.

[0128] Based on this, when faced with a deduplication request triggered by one or more business objects enjoying the target business service, the partition creation parameters for the distributed Bloom filter can be determined according to the cached target deduplication configuration information. Subsequently, the routing partition identifier is calculated by combining the business object identifier to determine the execution partition for the deduplication request of the business object.

[0129] In some implementations, the target deduplication configuration information corresponding to the target business service can be retrieved from the cache space to determine the partition creation parameters for the distributed Bloom filter based on the target deduplication configuration information. For example, step 102 may include: retrieving the pre-cached target deduplication configuration information corresponding to the target business service identifier from the cache space through the deduplication check service process; and determining the partition creation parameters when creating a distributed Bloom filter for the target business service based on the target deduplication configuration information.

[0130] Specifically, the deduplication check service refers to the integration of deduplication checking functionality into a business service through a deduplication detection toolkit. This deduplication check function relies on the business service and runs as a service process; that is, the deduplication check service is a service process itself. When a target business service has a deduplication check requirement, the deduplication check service can retrieve the target deduplication configuration information corresponding to the target business service identifier from the cache space. This target deduplication configuration information reflects the deduplication processing requirements of the target business service's operator. Based on this target deduplication configuration information, the partition creation parameters for the distributed Bloom filter created for the target business service can be determined. These partition creation parameters represent the number of Bloom filters created by the target business service within a single deduplication time period, which can be understood as the number of partitions within a single deduplication time period. This allows for subsequent routing calculations based on the number of partitions in the deduplication process to determine the execution partition corresponding to the deduplication request.

[0131] Using the above methods, the configuration information of the deduplication service of the corresponding target business service can be read from the cache space based on the request parameters carried in the deduplication request. This allows us to understand the partition creation parameters for the distributed Bloom filter, which can then be used to participate in the routing calculation of the partition for the Bloom filter, thereby determining the execution partition for the deduplication request.

[0132] 103. Determine the corresponding routing partition identifier based on the receiving object identifier and partition creation parameters, and select at least one target partition corresponding to the deduplication time range in the distributed service system based on the target business service identifier and routing partition identifier.

[0133] In this embodiment of the application, after obtaining the partition creation parameters of the distributed Bloom filter for the target business service, the allocation basis for the deduplication request can be calculated. Specifically, since the distributed Bloom filters of the target business service are used to manage fixed business objects, that is, each Bloom filter manages and stores the data of a fixed group of business objects, the routing partition identifier can be calculated by combining the receiving object identifier and the partition creation parameters, so as to select the target partition for performing the current deduplication processing according to the routing partition identifier and the deduplication time.

[0134] Prior to this, for each business service, a corresponding Bloom filter needs to be created in the distributed service system based on the corresponding deduplication configuration information. Thus, the distributed service system contains pre-created partitions for different business services' deduplication configuration information, and a Bloom filter associated with each partition. Specifically, this distributed service system can be a distributed system for deduplication services (or deduplication storage services), such as a Redis service cluster containing multiple Redis service nodes. At any given time, for the deduplication configuration information of each business service, multiple corresponding partitions and a corresponding Bloom filter for each partition can be created on the Redis service nodes of this Redis service cluster for subsequent deduplication checks.

[0135] It's important to note that Bloom filters have a time-dependent characteristic. Specifically, for business services, the deduplication configuration information stipulates that a corresponding number of partitions must be created in the distributed service system for each deduplication period, and a Bloom filter must be created for each partition. That is, Bloom filters are divided according to the deduplication period, and different deduplication periods have the same number of partitions and Bloom filters. For the multiple partitions of the distributed Bloom filter corresponding to each deduplication period, these partitions can be numbered sequentially, so that each partition corresponds to a partition identifier. For example, assuming that 10,000 partitions need to be created in the distributed service system for the target business service, and each partition corresponds to a Bloom filter, then these 10,000 partitions are numbered sequentially as 1, 2, 3, 4, ... 10,000.

[0136] In some implementations, a Bloom filter corresponding to the corresponding partition can be created at any time at the corresponding creation address in the distributed service system based on the target deduplication configuration information. For example, before step 103, the following may be included:

[0137] (1) Load the target deduplication configuration information corresponding to the target business service identifier from the deduplication configuration information public terminal;

[0138] (2) Based on the target deduplication configuration information, determine the creation address, deduplication duration, and partition creation parameters of the distributed Bloom filter;

[0139] (3) Create a Bloom filter corresponding to the partition in the creation address of the distributed service system according to the target business service identifier, deduplication duration information and partition creation parameters.

[0140] The creation address can be the service address of the distributed service system or the node address of the service node. For example, a separate distributed service system can be set up for each business service, and a distributed service system can be used to deploy a distributed Bloom filter for a business service. In this case, the creation address can be the system address of the distributed service system.

[0141] For example, multiple business services can share a single distributed service system. This system creates and deploys distributed Bloom filters for multiple business services. In this case, the distributed service system contains multiple service nodes, and each service node can be used to deploy a Bloom filter for only one business service. By dividing a service node into multiple service shards, a corresponding Bloom filter is created on each shard. This allows for the creation of a distributed Bloom filter for a business service within a single service node, and the creation address is the node address of the service node.

[0142] The deduplication duration information can be the retention period requirement for deduplication checks of the corresponding business service. Specifically, when setting deduplication configuration information, business services can also set the duration of deduplication checks according to their needs. This deduplication duration information can be used to limit the retention time of Bloom filters. For example, assuming that the deduplication duration requirement of business service A is continuous deduplication for 3 days, that is, checking whether there is duplicate content within three days, then the Bloom filters of the past three days need to be stored in the distributed service system. For example, assuming that 10,000 partitions are created every day (deduplication period), and a corresponding Bloom filter is created for each partition, then the Bloom filters corresponding to the partitions for three days need to be retained, that is, the 10,000 Bloom filters corresponding to today's 10,000 partitions, the 10,000 Bloom filters corresponding to yesterday's 10,000 partitions, and the 10,000 Bloom filters corresponding to the day before yesterday's 10,000 partitions.

[0143] The partition creation parameter can be interpreted as the partition creation density within a deduplication time period, i.e., the number of partitions created in each deduplication time period. It should be noted that the "deduplication time period" mentioned in this embodiment refers to a unit of deduplication time, such as one hour, one day, or one week; no limitation is made here. This embodiment describes it as "one day as a deduplication time period." At midnight of a new day (e.g., 00:00:00), multiple partitions for that day can be created according to the partition creation parameter, and a Bloom filter corresponding to each partition can be created.

[0144] Specifically, taking a target business service as an example, when initially creating a distributed Bloom filter for the business service, the computer device can load the target deduplication configuration information entered by the operator of the target business service from the deduplication configuration information public terminal according to the target business service identifier. Then, based on the target deduplication configuration information, the creation address of the distributed Bloom filter for the new deduplication period is determined, i.e., the service storage address, the deduplication duration information, i.e., the duration of deduplication checks, and the partition creation parameters, i.e., the number (density) of partitions created in a deduplication period. Finally, according to the above information, multiple partitions corresponding to each deduplication period and a Bloom filter corresponding to each partition are created in the creation address of the distributed service system. In this way, the distributed service system contains multiple Bloom filters corresponding to multiple deduplication periods of a business service.

[0145] For example, taking the creation of Bloom filters for each deduplication period as an example, when a new deduplication period arrives, such as the start of a new day, a corresponding distributed Bloom filter can be created for each business service. For instance, taking a target business service as an example, the computer device can load the target deduplication configuration information entered by the operator of the target business service from the deduplication configuration information public terminal according to the target business service identifier. Then, based on the target deduplication configuration information, it determines the creation address, deduplication duration information, and partition creation parameters of the distributed Bloom filter for the new deduplication period. Finally, in the service node corresponding to the creation address in the distributed service system, multiple partitions are created according to the partition creation parameters, and a corresponding Bloom filter is created for each partition. It should be noted that the deduplication duration information can also be used to determine the target historical deduplication period for which the Bloom filter needs to be deleted. Specifically, the deduplication duration information limits the duration of the deduplication check. When a new deduplication period appears, after updating and creating a Bloom filter for a new deduplication period, the historical deduplication period with the longest time remaining from the current time among multiple deduplication periods is usually taken as the target historical deduplication period, and the Bloom filter for this target historical deduplication period needs to be deleted.

[0146] Following the above method, manage and create distributed Bloom filters for each business service in the distributed service system, and create a Bloom filter for each new deduplication period so that deduplication checks can be performed through the Bloom filter in the future.

[0147] In some implementations, Bloom filters have corresponding index information, which can be understood as the identifier of the Bloom filter, used to distinguish each Bloom filter. When creating a Bloom filter, the index can be used to detect whether a corresponding Bloom filter has been created in the distributed service system, so that if no Bloom filter is detected, a decision can be made to create the corresponding Bloom filter. For example, step (3) may include:

[0148] (3.1) Generate multiple filter indexes corresponding to at least one deduplication time period based on the target business service identifier, deduplication duration information and partition creation parameters;

[0149] (3.2) For each deduplication time period, the corresponding Bloom filters are sequentially detected in the creation address of the distributed service system to obtain the detection results;

[0150] (3.3) Based on the detection results, identify the target filter index that has not been created with the corresponding Bloom filter, and create the Bloom filter for the partition corresponding to the target filter index in the creation address of the distributed service system.

[0151] The filter index can be understood as an identifier for the corresponding Bloom filter, used to distinguish one Bloom filter from others. Specifically, a Bloom filter has a time characteristic, meaning it is divided according to the deduplication period. Therefore, Bloom filters can be associated with time information. Furthermore, within a deduplication period, each Bloom filter corresponds to a partition, so each Bloom filter is associated with a partition identifier. Additionally, each Bloom filter belongs to a corresponding business service, so each Bloom filter is associated with a business service identifier. Based on this, an index for a Bloom filter can be constructed using these three aspects: time attribute, business service identifier, and partition identifier. This index can represent a unique Bloom filter in the distributed service system. This satisfies the management needs of distributed Bloom filters for multiple business services in a distributed service system, distinguishing between different business services or multiple Bloom filters corresponding to the same business service, thus enhancing the accuracy of Bloom filter detection.

[0152] It's important to note that to avoid system instability and resource waste caused by repeatedly creating duplicate Bloom filters in the distributed service system, it's necessary to first check if a corresponding Bloom filter already exists in the distributed service system and then create any non-existent Bloom filters. Therefore, to improve the accuracy of Bloom filter detection, an index can be constructed based on three aspects: time attribute, business service identifier, and partition identifier. This index can then be used to detect whether a corresponding Bloom filter has been created in the distributed service system.

[0153] Specifically, after obtaining the creation address, deduplication duration information, and partition creation parameters based on the target deduplication configuration information, multiple filter indexes corresponding to at least one deduplication time period are generated according to the target business service identifier, deduplication duration information, and partition creation parameters. This deduplication time period is determined based on the deduplication duration information. For example, assuming the deduplication duration information is the past 3 days, with one day as a deduplication time period unit, multiple filter indexes corresponding to each day of the past three days need to be generated. Further, for each filter index, the existence of a corresponding Bloom filter is checked at the creation address (i.e., service address) in the distributed service system; that is, whether a Bloom filter for that filter index has been created. Finally, if no Bloom filter for that filter index is detected, a Bloom filter for the partition corresponding to the target filter index is created at the creation address in the distributed service system.

[0154] In some implementations, the index of each Bloom filter can be constructed based on three aspects: time attribute, business service identifier, and partition identifier. For example, step (3.1) may also include: determining the deduplication time period range based on the deduplication duration information; determining multiple partition identifiers to be created corresponding to a deduplication time period based on the partition creation parameters; and constructing multiple filter indexes corresponding to each deduplication time period for each deduplication time period in the deduplication time period range, in combination with the target business service identifier and each partition identifier to be created.

[0155] The deduplication time range can be a range of the number of deduplication time periods, reflecting the duration of a deduplication process. Specifically, during the deduplication process, if a deduplication time period is used as a unit of time, the deduplication time range can include one or more units of deduplication time periods. For example, a deduplication time range of three days would include three one-day periods (deduplication time periods).

[0156] The partition identifier to be created can be the identifier of the corresponding partition, which can be understood as the sequential number of the partition. Specifically, the partition creation parameter can represent the number of partitions to be created within a unit of deduplication time period, that is, the partition density within a unit of deduplication time period. Based on the number of partitions, the partition identifier to be created is generated. For example, suppose that 10,000 partitions need to be created in the distributed service system for the target business service. The 10,000 partitions are created and numbered sequentially as 1, 2, 3, 4, ... 10,000.

[0157] It should be noted that, in order to accurately detect whether a corresponding Bloom filter has been created in the distributed service system, after obtaining the creation address, deduplication duration information, and partition creation parameters, an index of the Bloom filter can be constructed based on three aspects: time attribute, business service identifier, and partition identifier. Specifically, the deduplication duration information is used to determine the deduplication period range to understand the duration of deduplication processing and to determine the time frame for deduplication. Then, based on the partition creation parameters, multiple partition identifiers to be created corresponding to a deduplication period are determined, and these partition identifiers are numbered sequentially. Finally, based on the deduplication period range and the current time information, multiple deduplication periods are determined, and for each deduplication period, multiple corresponding filter indexes are created by combining the target business service identifier and each partition identifier to be created.

[0158] For example, the 10,000 filter indices for deduplication period 1 of business service A are sequentially named "A-1-Deduplication Period 1", "A-2-Deduplication Period 1", "A-3-Deduplication Period 1", and so on, until "A-10000-Deduplication Period 1" is generated. Similarly, for deduplication period 2, the 10,000 filter indices are sequentially named "A-1-Deduplication Period 2", "A-2-Deduplication Period 2", "A-3-Deduplication Period 2", and so on. Thus, for a single business service, each deduplication period within the deduplication timeframe corresponds to multiple filter indices. Subsequently, the filter indices can be used to accurately detect whether a corresponding Bloom filter has been created in the distributed service system. If no Bloom filter is detected, a decision can be made to create the corresponding Bloom filter, avoiding duplicate creation and ensuring reliability.

[0159] In this embodiment, to handle high-concurrency deduplication requests from business services, after creating a distributed Bloom filter for the business service, each deduplication request is distributed to multiple distributed Bloom filters for processing based on the receiving object identifier. Specifically, a hash calculation is performed based on the receiving object identifier, and a routing partition identifier is calculated in conjunction with partition creation parameters. Further, at least one target partition is selected from the distributed service system based on the routing partition identifier, the target business service identifier, and the deduplication time range. This allows subsequent deduplication checks to be performed on the Bloom filters corresponding to the target partitions, ensuring a balanced distribution of deduplication requests to the corresponding partitions' Bloom filters for deduplication checks while handling high-concurrency deduplication requests from business services, thus providing reliability.

[0160] The partition identifier can be calculated based on the receiving object identifier and the partition creation parameters. This partition identifier is used to indicate the partition corresponding to a Bloom filter in the distributed Bloom filter. Therefore, after obtaining the routing partition identifier, the target partition can be selected by combining the routing partition identifier. Subsequently, deduplication checks need to be performed in the Bloom filter of the target partition corresponding to the routing partition identifier.

[0161] In some implementations, a hash calculation is performed based on the received object identifier, and then moduloed with the partition creation parameters to obtain the routing partition identifier. For example, step 103, "determining the corresponding routing partition identifier based on the received object identifier and the partition creation parameters," may include:

[0162] (103.a.1) Based on the partition creation parameters, determine the number of partitions created by the distributed Bloom filter of the target business service per unit time period;

[0163] (103.a.2) Perform a hash calculation on the identifier of the receiving object to obtain a candidate hash value;

[0164] (103.a.3) Take the modulo of the candidate hash value according to the number of partitions created to obtain the routing partition identifier.

[0165] Specifically, after obtaining the partition creation parameters, the number of partitions to be created for the target business service within a deduplication period can be determined based on these parameters. Since one partition corresponds to one Bloom filter, this can also be understood as the number of partitions created for the distributed Bloom filter. Simultaneously, using the business object (i.e., the receiving object) as the basis for deduplication requests, a hash calculation is performed on the receiving object to obtain the corresponding candidate hash value. Combined with the hash routing mechanism, based on the number of partitions created within a deduplication period, a modulo operation is performed on this candidate hash value to obtain the routing partition identifier. Subsequently, based on the calculated routing partition identifier, combined with the target business service identifier and the deduplication time range, the target partition associated with the receiving object identifier can be found. This allows for subsequent deduplication checks to be performed in the Bloom filter corresponding to the target partition, thus enabling partitioning of deduplication requests related to each business object (i.e., the receiving object) and improving deduplication efficiency.

[0166] In some implementations, the distributed service system includes pre-created partitions for different business services. Based on this, multiple pre-created partitions corresponding to the target business service identifier can be found from the distributed service system. At least one target partition is selected from the multiple pre-created partitions corresponding to the target business service identifier according to the routing partition identifier and the deduplication time range. For example, step 103, "selecting at least one target partition corresponding to the deduplication time range in the distributed service system according to the target business service identifier and the routing partition identifier," may include:

[0167] (103.b.1) Based on the target business service identifier, determine the partitions pre-created for the target business service in the distributed service system;

[0168] (103.b.2) Queue from the pre-created partitions multiple candidate partitions corresponding to the route partition identifier;

[0169] (103.b.3) From a plurality of candidate partitions, determine at least one target partition corresponding to at least one target deduplication period included in the deduplication time range.

[0170] It should be noted that in a distributed service system, Bloom filters are pre-created for the target business service. Each Bloom filter corresponds to a partition. Each partition is divided according to time and partition identifier. That is, each partition is associated with a corresponding deduplication time period (time information) and partition identifier. Each unit of time contains a partition corresponding to the partition identifier.

[0171] Based on this, pre-created partitions for the target business service can be found from the distributed service system according to the target business service identifier. These pre-created partitions contain multiple partitions corresponding to each deduplication time unit within the deduplication time range, and the multiple partitions corresponding to each deduplication time unit are numbered sequentially, such as according to the creation order, so that each partition corresponds to a partition identifier. Since the partition identifiers between different deduplication time units are independent of each other when numbering, different deduplication time units have the same partition identifier. Furthermore, multiple candidate partitions corresponding to the routing partition identifier can be filtered from these pre-created partitions according to the routing partition identifier. Finally, from these multiple candidate partitions, at least one target partition corresponding to at least one target deduplication time period included in the deduplication time range is determined.

[0172] In some implementations, the deduplication time range may include multiple target deduplication time periods, and multiple target partitions may be selected from multiple candidate partitions based on the target deduplication time periods. For example, step (103.b.3) may include: determining multiple target deduplication time periods corresponding to the deduplication time range; and selecting multiple target partitions corresponding to the multiple target deduplication time periods from multiple candidate partitions.

[0173] Specifically, Bloom filters are divided according to deduplication time periods and partition identifiers. For example, Bloom filters corresponding to 10,000 partitions per day can be understood as Bloom filters associated with time and identifier attributes, meaning there is an independent partition 1 (Bloom filter 1) for each day. Based on this, after determining multiple candidate partitions corresponding to the pre-created routing partition identifiers for the target business service in the distributed service system, multiple target deduplication time periods can be determined according to the deduplication time range. Furthermore, based on multiple candidate partitions, multiple target partitions corresponding to these multiple target deduplication time periods are selected. Thus, based on filtering multiple candidate partitions according to the target business service identifier and routing partition identifier, and further combining the deduplication time range set by the target business service for the receiving object, multiple target partitions are selected, realizing the filtering of target partitions according to personalized time ranges, so that deduplication checks can be performed in the Bloom filters of the target partitions, ensuring reliability.

[0174] Using the above method, after obtaining the partition creation parameters of the distributed Bloom filter for the target business service, we can start calculating the allocation basis for the deduplication processing request, namely the route partition identifier, so as to select the target partition based on the route partition identifier and the deduplication time range, so that deduplication checks can be performed in the Bloom filter of the target partition later.

[0175] 104. Send the received object identifier and deduplication information to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition can perform deduplication detection based on the received object identifier and deduplication information.

[0176] In this embodiment, after selecting a target partition in the distributed service system, the computer device can send the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition in the distributed service system. This enables the Bloom filter associated with each target partition to perform deduplication detection based on the receiving object identifier and deduplication information. In this way, the deduplication processing request triggered by the target business service can perform deduplication checks in the Bloom filter corresponding to the target partition, thereby improving the deduplication processing efficiency and accuracy of each business service.

[0177] In some implementations, step 104 may include: constructing a deduplication information index based on the received object identifier and deduplication information, and sending the deduplication information index to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the deduplication information index. It should be noted that constructing a deduplication information index from the received object identifier and deduplication information, and performing deduplication checks based on the deduplication information index, improves the convenience of deduplication checks and effectively avoids information loss during transmission to the Bloom filter of the target partition, thereby improving the accuracy and efficiency of deduplication checks.

[0178] It should be noted that the toolkit for deduplication checking in business service integration can provide at least two types of interfaces.

[0179] On the one hand, a deduplication check interface is provided. When a deduplication processing request is triggered through this deduplication check interface, the computer device sends the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the receiving object identifier and deduplication information, and obtains the deduplication check result fed back by the Bloom filter corresponding to each target partition. The deduplication check result feeds back the check result for the deduplication information index, such as whether it appears repeatedly within the deduplication time range.

[0180] On the other hand, a deduplication write interface is provided. When a deduplication processing request is triggered through this interface, the computer device sends the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition. This enables the Bloom filter associated with each target partition to perform deduplication detection based on the receiving object identifier and deduplication information, and obtain the deduplication check result returned by the Bloom filter corresponding to each target partition. If the deduplication check result indicates that no duplicate occurrences have been found within the deduplication time range, the deduplication information index is written to a target Bloom filter of the target partition corresponding to the current deduplication time period, thus recording the deduplication information index.

[0181] In some implementations, when it is detected that none of the Bloom filters corresponding to each target partition contain the receiving object identifier and the deduplication information index corresponding to the deduplication information, the receiving object identifier and the deduplication information index corresponding to the deduplication information are stored in the target Bloom filter corresponding to the target partition. For example, if the target interface type includes a deduplication write interface, after step 104, the following steps may be included: when the deduplication check results of the Bloom filters corresponding to each target partition are all found to be without the deduplication information index, determine the target Bloom filter for the target partition that matches the current time information; construct a deduplication information index based on the receiving object identifier and the deduplication information; send the deduplication information index to the target Bloom filter for storage, and push the target content corresponding to the deduplication information to the target receiving object corresponding to the receiving object identifier.

[0182] Specifically, after obtaining the deduplication check results from the Bloom filters corresponding to each target partition, if the deduplication check results determine that none of the Bloom filters corresponding to each target partition have a deduplication information index for the receiving object identifier and the deduplication information, this indicates that the Bloom filters corresponding to the target partitions within the deduplication time range do not have target content that duplicates the deduplication information. This target content can be determined according to the actual scenario; for example, it could be personalized recall materials, advertisements, or articles that do not duplicate the receiving object identifier, or advertisements or articles corresponding to the deduplication information that have not been pushed to the receiving object identifier. This is not limited here. Further, from the Bloom filters corresponding to multiple target partitions, the target Bloom filter mapped to the target partition corresponding to the current time is selected according to the current time information. A deduplication information index is constructed based on the receiving object identifier and the deduplication information, and this index is sent to the target Bloom filter for storage. Afterward, the target content corresponding to the deduplication information can be pushed to the target receiving object corresponding to the receiving object identifier, or personalized information recall can be performed on the target receiving object corresponding to the receiving object identifier based on the deduplication information. The specific operation can be determined according to the actual scenario.

[0183] By using the above methods, the receiving object identifier and deduplication information can be sent to the Bloom filter corresponding to each target partition in the distributed service system. This enables the Bloom filter associated with each target partition to perform deduplication checks based on the receiving object identifier and deduplication information, thereby improving the deduplication efficiency and accuracy of each business service.

[0184] As can be seen from the overall description of the embodiments of this application, the embodiments of this application can obtain a deduplication processing request triggered by a target business service. The deduplication processing request carries at least a receiving object identifier, a deduplication time range, deduplication information, and a target business service identifier. Based on the target business service identifier, the partition creation parameters of the distributed Bloom filter for the target business service are determined from the pre-cached target deduplication configuration information. Based on the receiving object identifier and the partition creation parameters, the corresponding routing partition identifier is determined, and based on the target business service identifier and the routing partition identifier, at least one target partition corresponding to the deduplication time range is selected in the distributed service system. The distributed service system includes partitions pre-created for deduplication configuration information of different business services, and a Bloom filter associated with each partition. The receiving object identifier and deduplication information are sent to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the receiving object identifier and the deduplication information.

[0185] Based on this, this application can obtain a deduplication request triggered by any target business service. This deduplication request contains various parameters defined according to a unified interface. Then, it reads the partition creation parameters of the target business service from the cache. These partition creation parameters can at least represent the number of partitions of the distributed Bloom filter created for the target business service within a deduplication time period. Thus, a routing partition identifier is generated by combining the receiving object identifier. Then, using the routing partition identifier and the deduplication time range as a filtering index, the target partition is selected in the distributed service system. Each target partition corresponds to the routing partition identifier in a deduplication time period within the deduplication time range. The distributed service system has multiple partitions and a corresponding Bloom filter for each target business service pre-created. Finally, by sending the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition, the application requests the Bloom filter corresponding to the target partition to perform deduplication checks. Therefore, compared to existing technologies that use database constraints, hash tables, or third-party services to perform deduplication checks, which cannot meet the deduplication needs of a wide range of business services, this application pre-creates corresponding distributed Bloom filters based on partition creation parameters for each business service. Each Bloom filter corresponds to a partition, and each partition is divided according to the deduplication period and partition identifier to meet the deduplication detection needs of different business services. Furthermore, each business service can carry various parameters defined by a unified interface in its request to request the Bloom filter of the target partition corresponding to each business service to perform deduplication detection. In this way, unified management and concurrent processing of deduplication processing for each business service are achieved, improving the efficiency and accuracy of deduplication processing for each business service.

[0186] Based on the methods described in the above embodiments, the following examples will provide further detailed explanations.

[0187] Figure 3 This is a schematic flowchart of another step in the data processing method provided in the embodiments of this application. For ease of understanding, the embodiments of this application are combined with... Figure 3 Describe it.

[0188] In this embodiment, the data processing method is described using an example where the data processing method is executed by a server. The specific flow of the data processing method is as follows:

[0189] 201. Load the target deduplication configuration information corresponding to the target business service identifier from the deduplication configuration information public terminal, and determine the creation address, deduplication duration information, and partition creation parameters of the distributed Bloom filter based on the target deduplication configuration information.

[0190] The server in this application embodiment can be a distributed system type server, including multiple sub-servers, such as a deduplication configuration information public terminal for storing deduplication configuration information entered by each business service, a business server for running business services, a service cluster of Bloom filters corresponding to business services, a management server for managing and creating Bloom filters for business services, etc. All of the above can be understood as computer devices in this application embodiment.

[0191] To meet the diverse deduplication needs of multiple business services, operators of each service input their deduplication configuration information into the deduplication configuration entry page on the public deduplication configuration information platform, thus storing this information. At this point, it is necessary to synchronize and cache the deduplication configuration information of each service so that when a service subsequently has a deduplication requirement, it can generate a deduplication check request based on the cached configuration information and send the request to the designated Bloom filter.

[0192] Specifically, the computer equipment can load the target deduplication configuration information entered by the operator of the target business service from the public terminal for deduplication configuration information, according to the target business service identifier. Then, based on the target deduplication configuration information, it determines the creation address of the distributed Bloom filter for the new deduplication period, i.e., the service storage address, the deduplication duration information, i.e., the duration of deduplication checks, and the partition creation parameters, i.e., the number of partitions created in a deduplication period. In this way, the deduplication requirements of the target business service are obtained.

[0193] 202. Based on the target business service identifier, deduplication duration information, and partition creation parameters, generate multiple filter indexes corresponding to at least one deduplication time period. For each multiple filter index corresponding to a deduplication time period, sequentially detect the corresponding Bloom filter in the creation address of the distributed service system to obtain the detection results.

[0194] In this embodiment, to avoid system disorder and resource waste caused by repeatedly creating duplicate Bloom filters in the distributed service system, it is necessary to first detect whether a corresponding Bloom filter has been created in the distributed service system, and then create any non-existent Bloom filters. Based on this, to improve the detection accuracy of Bloom filters, an index of Bloom filters can be constructed based on three aspects: time attribute, business service identifier, and partition identifier. This index can then be used to detect whether a corresponding Bloom filter has been created in the distributed service system.

[0195] 203. Based on the detection results, identify the target filter indexes that do not have corresponding Bloom filters, and create the Bloom filter for the partition corresponding to the target filter index in the creation address of the distributed service system.

[0196] In this embodiment, if the detection results determine that no Bloom filter corresponding to the filter index has been created in the distributed service system, then a Bloom filter for the partition corresponding to the target filter index is created in the creation address of the distributed service system. This completes the creation of the distributed Bloom filter for the target business service, enabling subsequent deduplication checks to be performed on the distributed Bloom filter of the target business service based on the received object identifier and deduplication information.

[0197] 204. Deduplication requests triggered by obtaining the target business service.

[0198] In order to unify the management of deduplication requirements of various business services and simplify the deduplication process of each business service, this application provides a toolkit for deduplication checking. Each business service integrates deduplication checking functionality through the toolkit. Based on the parameter types defined in the interface of the toolkit, when any business service requires deduplication checking functionality, the relevant parameters of that business service are collected and obtained for the implementation of deduplication checking.

[0199] It should be noted that the business service integrates deduplication functionality through a deduplication check toolkit (SDK). When a target business service needs to perform deduplication processing during operation, such as during information push, it calls the target interface for the deduplication function, providing relevant request parameters according to the parameter types defined in the target interface. These parameters include, for example, the receiving object identifier, deduplication information, deduplication time range, and the target business service identifier. Based on these parameters, a deduplication request is generated, triggering the deduplication process. This enables unified management of deduplication requirements across multiple business services. Through the SDK's interface, business services only need to provide the corresponding parameter interfaces according to their defined parameters, simplifying the deduplication process.

[0200] 205. Based on the target service identifier, determine the partition creation parameters of the distributed Bloom filter for the target service from the pre-cached target deduplication configuration information.

[0201] In this embodiment, after detecting a deduplication request triggered by the deduplication check function, the configuration information of the deduplication service for the corresponding target business service can be read from the cache space according to the request parameters carried in the deduplication request. Specifically, based on the target business service identifier in the request parameters, the partition creation parameters of the distributed Bloom filter created for the deduplication configuration requirements of the target business service can be obtained from the cache space, so as to participate in the routing calculation of the partition of the Bloom filter and determine the execution partition of the deduplication request.

[0202] Specifically, the deduplication check service refers to the integration of deduplication check functionality into a business service through a deduplication detection toolkit. This deduplication check function relies on the business service and runs as a service process; that is, the deduplication check service is a service process itself. When a target business service has a deduplication check requirement, the deduplication check service can retrieve the target deduplication configuration information corresponding to the target business service identifier from the cache space. This target deduplication configuration information reflects the deduplication processing requirements of the target business service's operator. Based on this target deduplication configuration information, the partition creation parameters for the distributed Bloom filter created for the target business service can be determined. These partition creation parameters represent the number of Bloom filters created by the target business service within a single deduplication time period, which can be understood as the number of partitions within a single deduplication time period. This allows for subsequent routing calculations based on the number of partitions used in the deduplication process.

[0203] 206. Determine the corresponding routing partition identifier based on the receiving object identifier and partition creation parameters, and select at least one target partition corresponding to the deduplication time range in the distributed service system based on the target business service identifier and routing partition identifier.

[0204] In this embodiment of the application, after obtaining the partition creation parameters of the distributed Bloom filter for the target business service, the allocation basis for the deduplication request can be calculated. Since the distributed Bloom filters of the target business service are used to manage fixed business objects, that is, each Bloom filter manages and stores the data of a fixed group of business objects, the routing partition identifier can be calculated by combining the receiving object identifier and the partition creation parameters.

[0205] Specifically, after obtaining the partition creation parameters, the number of partitions to be created for the target business service within a deduplication period can be determined based on these parameters. Since one partition corresponds to one Bloom filter, this can also be understood as the number of partitions created for the distributed Bloom filter. Simultaneously, using the business object (i.e., the receiving object) as the basis for deduplication requests, a hash calculation is performed on the receiving object to obtain the corresponding candidate hash value. Combined with the hash routing mechanism, based on the number of partitions created within a deduplication period, a modulo operation is performed on this candidate hash value to obtain the routing partition identifier. Subsequently, based on the calculated routing partition identifier, combined with the target business service identifier and the deduplication time range, the target partition associated with the receiving object identifier can be found. This allows for subsequent deduplication checks to be performed in the Bloom filter corresponding to the target partition, thus enabling partitioning of deduplication requests related to each business object (i.e., the receiving object) and improving deduplication efficiency.

[0206] Furthermore, the target partition for the target business service is selected based on the routing partition identifier and the deduplication time range. It should be noted that in the distributed service system, each pre-created Bloom filter for the target business service corresponds to a partition, and each partition is associated with a corresponding deduplication time period (time information) and a partition identifier. Each unit of time contains one partition corresponding to the partition identifier. Based on this, the pre-created partitions for the target business service can be found in the distributed service system according to the target business service identifier. Then, multiple candidate partitions corresponding to the routing partition identifier can be filtered from these pre-created partitions. Finally, from these multiple candidate partitions, at least one target partition corresponding to at least one target deduplication time period included in the deduplication time range is determined. This allows the target partition for subsequent deduplication processing to be performed in the Bloom filter corresponding to that target partition.

[0207] 207. Send the received object identifier and deduplication information to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition can perform deduplication detection based on the received object identifier and deduplication information.

[0208] In this embodiment, after selecting a target partition in the distributed service system, the computer device can send the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition in the distributed service system. This enables the Bloom filter associated with each target partition to perform deduplication detection based on the receiving object identifier and deduplication information. In this way, the deduplication processing request triggered by the target business service can perform deduplication checks in the Bloom filter corresponding to the target partition, thereby improving the deduplication processing efficiency and accuracy of each business service.

[0209] When a deduplication processing request is triggered through the deduplication check interface, the computer device sends the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition. This enables the Bloom filter associated with each target partition to perform deduplication detection based on the receiving object identifier and deduplication information, and obtain the deduplication check result fed back by the Bloom filter corresponding to each target partition. The deduplication check result feeds back the check result for the deduplication information index, such as whether it appears repeatedly within the deduplication time range.

[0210] When a deduplication request is triggered through this deduplication write interface, the computer device sends the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition. This allows the Bloom filter associated with each target partition to perform deduplication detection based on the receiving object identifier and deduplication information, and obtain the deduplication check result returned by the Bloom filter corresponding to each target partition. If the deduplication check result indicates that no duplicate occurrences have been found within the deduplication time range, the deduplication information index is written to a target Bloom filter of the target partition corresponding to the current deduplication time period, thus recording the deduplication information index.

[0211] To facilitate understanding of the embodiments of this application, specific application scenario examples will be used to describe the embodiments of this application. Specifically, the application scenario example will be described by performing the above steps 201-207.

[0212] It should be noted that this data processing method is applicable to deduplication checking services provided for operations of various business tenants. For example, taking the information push platform as an example, deduplication functionality is used in multiple stages of the push chain, such as number package generation, personalized recall, and information push. That is, a business service can have one or more business scenarios. To uniformly manage the deduplication checking of these business scenarios, a unified deduplication interface has been designed, along with a complete management backend and services. It supports massive business access through business configuration, and the system has good performance, disaster recovery, and cost considerations. The following is a detailed introduction to the data processing method using an example of this data processing scenario:

[0213] I. A brief introduction to this data processing scenario is as follows:

[0214] Since the push platform may connect to tenants of multiple business services, the push platform needs to provide deduplication checking services for each connected business service tenant.

[0215] Related technologies generally employ database constraint techniques. By setting unique constraints or primary keys at the database level, duplicate data can be prevented from being inserted, thus achieving deduplication. Alternatively, by using hash tables (such as Java's HashSet or Python's Set) in memory to store processed elements, it is possible to determine whether an element is a duplicate in O(1) time complexity. Or, deduplication checks can be performed using data type deduplication techniques provided by third-party services. However, the deduplication requirements of each business service are generally different, and different deduplication requirements are also different. Related technologies are only applicable to some basic deduplication scenarios and cannot meet the complex situation of multiple business scenarios corresponding to each business service under multiple business services.

[0216] Unlike related technologies, this data processing scenario example is as follows: A deduplication checking technology based on multi-service tenants is proposed. Specifically, considering the factors of multi-service tenants and multi-service scenarios, a unified deduplication interface is designed for this factor. At the same time, a complete set of management backends and services are provided. Through business configuration, it supports the access of massive business. Thus, during the application process, business services only need to provide the corresponding parameter information according to the unified deduplication interface to initiate deduplication checking requests, without having to pay attention to the details of the backend storage implementation. It is convenient and meets the deduplication needs of a wide range of business service scenarios. At the same time, the system has good performance, disaster recovery and cost considerations.

[0217] II. The specific implementation process of this data processing scenario example is as follows:

[0218] Figure 4 A framework example diagram of the deduplication check service provided in the embodiments of this application. See also... Figure 4 As shown, the architecture of the deduplication check service corresponding to this data processing scenario example includes business tenants, a common configuration terminal for multiple business tenants, a deduplication server, and a management terminal.

[0219] The configuration public terminal for this multi-service tenant supports configuration for multiple service tenants. The operators of the relevant service tenants can enter deduplication configurations and rules according to the deduplication requirements of their respective service, so as to be used for the construction of their respective deduplication services in the future.

[0220] The management end periodically reads the deduplication configuration information of the multi-service tenants from the common configuration terminal of the multi-service tenants in order to pre-create deduplication storage in the deduplication service.

[0221] The deduplication service includes an SDK and a backend deduplication storage. When a business scenario triggers a deduplication check requirement, it calls the unified interface provided by the SDK, namely the deduplication call interface. The SDK provides the relevant parameters according to its defined parameters to generate a deduplication request, which is then accessed via a Remote Procedure Call (RPC) request to the backend deduplication storage. Furthermore, this deduplication storage provides core deduplication functions such as data storage, data querying, and updating, performing deduplication checks for the corresponding business logic.

[0222] Figure 5 This is a structural example diagram of the deduplication check service system provided in the embodiments of this application. Combined with... Figure 4 and Figure 5 As shown, the deduplication service system structure includes business server A for business service A, business server B for business service B, a terminal (which may have a client installed), a Redis service cluster for the "push business advertisement a" scenario, a Redis service cluster for the "personalized recall" scenario, a common configuration terminal for multiple business tenants, a management terminal, and so on.

[0223] The configuration public terminal for multi-service tenants includes the deduplication configuration information entered by the operator of service A.

[0224] The management system obtains the deduplication configuration information of each business tenant from the common configuration client of multiple business tenants, and creates the deduplication storage (i.e., Redis service cluster) corresponding to the business tenant according to the deduplication configuration information. For example, the management system creates a Redis service cluster for the "push business advertisement a" scenario and a Redis service cluster for the "personalized recall" scenario of business service A based on the deduplication configuration information of business tenant A.

[0225] Among them, business service A integrates a deduplication SDK. The deduplication SDK periodically pulls the deduplication configuration information of business tenant A from the common configuration terminal of multiple business tenants and caches the pulled deduplication configuration information. User 1 (id=1) and user 2 (id=2) can access business service A through their respective terminals (such as the laptop in the example figure). Business service A provides relevant parameter information according to the defined parameter types of the integrated deduplication SDK to generate a deduplication check request and send the deduplication check request to the deduplication storage (i.e., the Redis service cluster).

[0226] The deduplication storage (i.e., the Redis service cluster) includes multiple Bloom filters for deduplication requirements of deduplication configuration information. Deduplication checking refers to performing deduplication checks within the Bloom filters.

[0227] It should be noted that the management server and the business server can be sub-servers belonging to the same distributed system; this is not a limitation here.

[0228] To facilitate understanding of this data processing scenario example, the details of each deduplication check service will be introduced below:

[0229] When dealing with extremely large amounts of data, Bloom filters can be used for deduplication checks, achieving a balance between data capacity and cost. However, for push platforms, single-machine Bloom filters cannot meet the requirements of high concurrency and scalability. This data scenario example uses Redis's distributed Bloom filter to store deduplicated data. The following section provides a detailed description of the deduplication service for multi-service tenants.

[0230] (1) Multi-service tenant configuration

[0231] Figure 6 This is an example diagram illustrating a scenario where the deduplication SDK provided in this application loads deduplication configuration information. Combined with... Figure 6As shown, the configuration common terminal for multiple service tenants provides a web management console (i.e., Remote ConfigCMS) for each service tenant to enter deduplicated configuration information. Figure 6 The system stores the deduplication configuration information entered by each business tenant. For example, the deduplication configuration information for business tenant A is represented as "Business A: Business ID, Number of Bloom filters, Deduplication storage address, ...", and the deduplication configuration information for business tenant B is represented as "Business B: Business ID, Number of Bloom filters, Deduplication storage address, ...". This deduplication configuration information can be remotely stored in the configuration common terminal of multiple business tenants in the form of a registry.

[0232] In addition, the multi-service tenant configuration public terminal provides a remote configuration service (Remote Config Server), and the deduplication SDK includes a remote configuration SDK (Remote Config SDK) so that the deduplication SDK of business services can dynamically load the deduplication configuration information stored remotely in the multi-service tenant configuration public terminal. Specifically, the deduplication SDK uses a polling (periodic retrieval) mechanism to obtain multi-tenant configuration data and cache it in local memory and local file system. Businesses can directly query the cache to obtain data, thereby supporting hot configuration updates and fault recovery.

[0233] When entering their respective deduplication configuration information, each business tenant can set their own deduplication configuration information according to the actual situation. For example, the deduplication configuration information may include the following:

[0234]

[0235] Figure 7 This is an example diagram of the page for entering deduplication configuration information provided in an embodiment of this application. Combined with... Figure 7 As shown, business tenants can enter deduplication configuration information on the deduplication configuration information entry page. This deduplication configuration information includes business identifier (id), business name, maximum deduplication days, false positive rate, peak QPS, North Star address (i.e., deduplication storage service address), effective status, creator (such as business tenant or business tenant's administrator), modification time, creation time, etc. Enter the above information items on the deduplication configuration information entry page to create their own deduplication configuration information.

[0236] (2) Management terminal

[0237] The management service periodically reads the deduplication configuration information of each business tenant to perform pre-creation (building) and check (check) of deduplication storage.

[0238] Figure 8This is a flowchart illustrating the creation process of a Bloom filter provided in this application embodiment. Specifically, first, a Bloom filter key is generated according to the format "{Business ID}_{Partition ID}_{Date}". At regular intervals, the configuration of each business tenant is retrieved from the common configuration client for multiple business tenants, and the creation of the Bloom filter is performed sequentially. Combined with... Figure 8 As shown, the creation process is as follows:

[0239] (A.1) Obtain the deduplication configuration information of a business tenant.

[0240] (A.2) Perform pre-creation and check of the Bloom filter based on the number of partitions in the deduplication configuration information. Specifically, first check if it exists, and if it does not exist, create it.

[0241] (A.3) Create a Bloom filter partition. A Bloom filter partition can include associated partition attributes, for example, the partition attribute can be {BF.RESERVE(key)(error_rate)(capacity)}.

[0242] (A.4) Check each Bloom filter partition. Specifically, you can perform the check according to {BF.INFO(key)} to get the check results.

[0243] (A.5) Determine whether all Bloom filter partitions have been created based on the inspection results. If not, repeat steps (A.2) to (A.5) until all Bloom filter partitions have been created based on the inspection results, and proceed to step (A.6).

[0244] (A.6) Determine if all business tenants' Bloom filter partitions have been created. If not, identify the next business tenant for which a Bloom filter partition needs to be created, and complete the creation of the next tenant's Bloom filter partition according to steps (A.1) to (A.5). If yes, end the creation process.

[0245] It's important to note that Bloom filters have an active status, determined by time. The validity period is determined based on the deduplication time information, typically in days (24 hours). Bloom filters expire at 00:00 on the second day after their expiration date. For example, if a business tenant needs to perform deduplication for 3 days, and a Bloom filter becomes active on [date], it would generally be a Bloom filter created the previous day (i.e., [date]), with an active period from [date] to [date] 13. This Bloom filter would expire at 00:00 on [date] 14, typically with a 5-minute offset. For instance, it might be cleared between 00:00:00 and 00:00:05 on [date] 14, thus preventing concentrated Bloom filter expiration and impacting the deduplication service performance.

[0246] (3) Deduplication storage

[0247] Figure 9 This is an example diagram of the deduplication storage architecture provided in an embodiment of this application. Combined with... Figure 9 As shown, the deduplication storage can include virtual partitions in the architecture. Each virtual partition has multiple partitions, a corresponding Bloom filter for each partition, and multiple Bloom filters evenly distributed across various Redis shards. These multiple Redis shards form a Redis cluster.

[0248] Among them, Bloom filters have small footprint, fast query speed, and support large amounts of data, making them suitable for deduplication checks within an acceptable error rate range. For example, deduplication checks in push middleware services can rely on Bloom filters.

[0249] The Bloom filter is evenly distributed across the shards of the Redis cluster. Bloom filters on Redis shards offer advantages such as high performance, distributed support, ease of integration, data persistence, dynamic adjustment, and open-source community support. By providing deduplication services through distributed Bloom filters in the Redis cluster, the limitations of single-instance Redis memory capacity and peak QPS (queries per second) are overcome, offering better horizontal scalability and disaster recovery capabilities.

[0250] Each Bloom filter corresponds to a partition. Specifically, since multiple Bloom filters are physically divided and evenly distributed across the Redis cluster, each Bloom filter can be assigned to a partition to facilitate the distribution of deduplication checks. Multi-partition routing determines which partition's Bloom filter a deduplication check request should be sent to based on the sharding key and sharding strategy (e.g., range sharding or hash sharding). The system achieves load balancing, improves query performance, and enhances scalability through multi-shard routing. For example, assuming a Bloom filter requires 500 gigabytes (G) of deduplication space and is divided into 10,000 partitions, each partition occupies only 50MB of space, and each partition is responsible for 1 / 10,000 of the deduplication checks.

[0251] The next step is to route to the target partition in the virtual partition based on the shard key, so that deduplication checks can be performed on the Bloom filter corresponding to the target partition.

[0252] (4) Deduplication SDK

[0253] The deduplication SDK provides a unified deduplication interface. By introducing the SDK, businesses can initiate deduplication requests without needing to concern themselves with the details of the backend storage implementation. Specifically, it includes a check interface and a write interface. The check interface supports deduplication checks over multiple days, that is, checking whether a deduplication information index (deduplication key) has appeared within N days. The write interface can be used to write the deduplication information index into the latest (current) Bloom filter and return the result of whether it has repeated within N days. In other words, the write interface can perform deduplication checks and write operations simultaneously.

[0254] The parameters defined in the interface can be represented as follows:

[0255]

[0256] The business services integrate the corresponding functions of the deduplication SDK by introducing the deduplication SDK. The logic of the deduplication SDK is as follows:

[0257] First, the deduplication key, sharding key (route key), and business identifier (id) are specified through the SDK interface. The SDK retrieves business configuration information from the cache space using the business ID, and calculates the partition identifier (id) based on the route key and the number of partitions in the business configuration information. Then, based on the specified deduplication days, the current day, and the partition identifier, the SDK routes to the target partition within the virtual partition to find the corresponding target Bloom filter. Finally, a deduplication check request is sent to the target Bloom filter corresponding to each target partition. The target Bloom filter performs a duplicate check and returns the deduplication result.

[0258] Figure 10 This is an example diagram of a partitioned routing scenario using a Bloom filter provided in an embodiment of this application. Combined with... Figure 10 As shown, for example, suppose the request specifies the number of days (i.e., the deduplication time range) N, the current day's time T, and the partition identifier id. Specifically, when routing to the corresponding target partition, the target partition corresponding to the partition identifier id within the time range [T, T-1, ... TN-1] is determined. For example, assuming N = 7 and the partition identifier is 1, then the target partition 1 is found among the partitions in the past 7 days, such as partition 1 on day 1, partition 1 on day 2, partition 1 on day 3, partition 1 on day 4, partition 1 on day 5, partition 1 on day 6, and partition 1 on day 7. Subsequently, deduplication can be performed on the "deduplication key" in the Bloom filter corresponding to the target partitions found above.

[0259] Figure 11 This is a logical flowchart of the deduplication SDK provided in this application embodiment. The request parameters defined in the SDK interface include the following: "Business ID: A", "Deduplication key: xxxxxxx (hash of deduplication information index)", "Sharding key (RouteKey): xxxx", and "Deduplication days (i.e. deduplication time range): 7 days".

[0260] The algorithm checks for duplicates by performing a `bf.exists` operation on the deduplication key in the Bloom filter for the most recent 7 days; simultaneously, it updates the deduplication key in the Bloom filter for the current day. Specifically, it combines... Figure 11 As shown, the logical flow is as follows:

[0261] (B.1) Start sending a deduplication check request to the deduplication service. The deduplication check request includes the business identifier "A", the deduplication key, and the sharding key "routekey".

[0262] (B.2) Calculate the partition identifier (i.e., partition ID) based on the sharding key "routekey".

[0263] (B.3) Generate a Bloom filter key based on the business identifier "A", the partition identifier and the deduplication time, such as "A-partition identifier-deduplication time period 1", "A-partition identifier-deduplication time period 2", "A-partition identifier-deduplication time period 7", etc.

[0264] (B.4) In the target Bloom filter corresponding to the Bloom filter key, perform a deduplication check on the deduplication key. For example, in the target Bloom filter corresponding to "A-Partition Identifier-Deduplication Time Period 1", performing a deduplication check on the deduplication key can be represented as "bf.exists A-Partition Identifier-Deduplication Time Period 1xxxxxxx". Figure 11 As shown, assuming a deduplication check is performed on the deduplication key in the Bloom filter for the current day, represented as "bf.add A-partition identifier-deduplication period 7xxxxxxx", the deduplication check operation for the current day is the same as and repeats the deduplication check operation for the previous 6 days. This means that a deduplication check is performed on the Bloom filter for the most recent 7 days for the deduplication key. End.

[0265] The deduplication SDK offers the following advantages: First, businesses only need to focus on using the interface and specifying core parameters such as business ID, deduplication key, routing key, and deduplication days when sending deduplication check requests, without needing to concern themselves with implementation details. Second, the SDK periodically synchronizes the business configuration information (i.e., deduplication configuration information) of each business tenant from the common configuration client of multiple business tenants to update the business deduplication configuration. Furthermore, it can also detect key metrics and configure corresponding alarms.

[0266] Figure 12 This is an example diagram illustrating the performance metrics monitoring of the deduplication checking service provided in this application embodiment. For example, combined with... Figure 12 As shown, the core metrics of the deduplication check service can include time consumption (exec_cost), number of successful requests (exec_succ), number of failed requests (exec_fail), duplicate deduplication information (key_exist), and unique deduplication information (key unique). The data of one or more of the above metrics in each deduplication period are statistically analyzed and displayed in a statistical chart.

[0267] (4) Monitor the capacity of the Bloom filter.

[0268] Regularly retrieve configurations from multiple service tenants and periodically monitor the capacity of each Bloom filter in the distributed service system. When the capacity reaches a certain threshold (e.g., 80% of the total capacity), trigger an alarm.

[0269] Figure 13 This is an example diagram illustrating the capacity monitoring information of a Bloom filter provided in an embodiment of this application. It is assumed that an alarm is issued when the capacity of the Bloom filter exceeds 80%, combined with... Figure 13As shown in the figure, the Bloom filter corresponding to "104_3534_20230518" has a capacity exceeding 80%. Specifically, this means that the total capacity of the Bloom filter corresponding to partition 3534 of service 104 on May 18, 2023 is "cap: 500000", and the currently used capacity is "num: 487835". Similarly, the total capacity of the Bloom filter corresponding to partition 3533 of service 104 on May 18, 2023 is "cap: 500000", and the currently used capacity is "num: 487835". This is an alarm message for the capacity monitoring of the two Bloom filters of service 104.

[0270] By executing the above data processing scenario examples, the following effects can be achieved: For various business scenarios of different business tenants, corresponding distributed Bloom filter clusters can be created respectively, so that each distributed Bloom filter cluster can meet the capacity requirements of deduplication information indexing for the corresponding business scenario, and meet the deduplication configuration requirements of different business tenants and different business scenarios under each business tenant. It supports the access of massive business tenants and business scenarios, meets the different deduplication requirements of multiple business tenants and multiple business scenarios, realizes unified management of different deduplication requirements of multiple business tenants and multiple business scenarios, and improves the efficiency and accuracy of deduplication checks for various business services.

[0271] As described above, this application embodiment can obtain a deduplication request triggered by any target business service. This deduplication request contains various parameters defined according to a unified interface. Then, the partition creation parameters of the target business service are read from the cache. These partition creation parameters can at least represent the number of partitions of the distributed Bloom filter created for the target business service within a deduplication time period. Thus, a routing partition identifier is generated by combining the receiving object identifier. Then, the routing partition identifier and the deduplication time range are used as filtering indexes to select target partitions in the distributed service system. Each target partition corresponds to a routing partition identifier in a deduplication time period within the deduplication time range. The distributed service system has multiple partitions and Bloom filters corresponding to each partition pre-created for the target business service. Finally, the receiving object identifier and deduplication information are sent to the Bloom filter corresponding to each target partition to request the Bloom filter corresponding to the target partition to perform deduplication checks. Therefore, compared to existing technologies that use database constraints, hash tables, or third-party services to perform deduplication checks, which cannot meet the deduplication needs of a wide range of business services, this application pre-creates corresponding distributed Bloom filters based on partition creation parameters for each business service. Each Bloom filter corresponds to a partition, and each partition is divided according to the deduplication period and partition identifier to meet the deduplication detection needs of different business services. Furthermore, each business service can carry various parameters defined by a unified interface in its request to request the Bloom filter of the target partition corresponding to each business service to perform deduplication detection. In this way, unified management and concurrent processing of deduplication processing for each business service are achieved, improving the efficiency and accuracy of deduplication processing for each business service.

[0272] For details on the implementation of each of the above steps, please refer to the previous examples, which will not be repeated here.

[0273] To facilitate better implementation of the data processing method provided in the embodiments of this application, the embodiments of this application also provide an apparatus based on the above-described data processing method. The meanings of the terms used are the same as in the data processing method described above, and specific implementation details can be found in the descriptions within the method embodiments.

[0274] Please see Figure 14 , Figure 14 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. The data processing device is integrated into the computer equipment of this application, and the data processing device may include a first acquisition unit 401, a determination unit 402, a selection unit 403, and a sending unit 404.

[0275] The acquisition unit 401 is used to acquire the deduplication processing request triggered by the target business service. The deduplication processing request carries at least the receiving object identifier, the deduplication time range, the deduplication information, and the target business service identifier.

[0276] The determining unit 402 is used to determine the partition creation parameters of the distributed Bloom filter for the target business service from the pre-cached target deduplication configuration information based on the target business service identifier.

[0277] The selection unit 403 is used to determine the corresponding routing partition identifier based on the receiving object identifier and the partition creation parameters, and to select at least one target partition corresponding to the deduplication time range in the distributed service system based on the target service identifier and the routing partition identifier.

[0278] The distributed service system includes pre-created partitions for deduplication configuration information of different business services, as well as Bloom filters associated with each partition.

[0279] The sending unit 404 is used to send the receiving object identifier and deduplication information to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition can perform deduplication detection based on the receiving object identifier and deduplication information.

[0280] In some embodiments, the selected unit 403 is further configured to:

[0281] Based on the target business service identifier, determine the partitions pre-created for the target business service in the distributed service system;

[0282] Query multiple candidate partitions corresponding to the route partition identifier from the pre-created partitions;

[0283] From multiple candidate partitions, determine at least one target partition corresponding to at least one target deduplication time period included in the deduplication time range.

[0284] In some embodiments, the selected unit 403 is further configured to:

[0285] Determine the multiple target deduplication time periods corresponding to the deduplication time range;

[0286] From multiple candidate partitions, select multiple target partitions corresponding to multiple target deduplication time periods.

[0287] In some embodiments, the selected unit 403 is further configured to:

[0288] Based on the partition creation parameters, determine the number of partitions created by the distributed Bloom filter of the target business service per unit time period;

[0289] Perform a hash calculation on the identifier of the receiving object to obtain a candidate hash value;

[0290] The candidate hash value is moduloed according to the number of partitions created to obtain the routing partition identifier.

[0291] In some embodiments, the determining unit 402 is further configured to:

[0292] By checking the deduplication service process, the pre-cached target deduplication configuration information corresponding to the target business service identifier is found in the cache space;

[0293] Based on the target deduplication configuration information, determine the partition creation parameters when creating a distributed Bloom filter for the target business service.

[0294] In some embodiments, the data processing apparatus further includes a loading unit for:

[0295] By checking the deduplication service process, the target deduplication configuration information corresponding to the target business service identifier is loaded from the deduplication configuration information public terminal.

[0296] Among them, the deduplication configuration information public terminal contains at least one business service for deduplication configuration information input for deduplication checks;

[0297] The loaded target deduplication configuration information is cached in the cache space.

[0298] In some embodiments, the data processing apparatus further includes a creation unit, which is also used for:

[0299] Load the target deduplication configuration information corresponding to the target business service identifier from the deduplication configuration information public terminal;

[0300] Based on the target deduplication configuration information, determine the creation address, deduplication duration, and partition creation parameters of the distributed Bloom filter;

[0301] Based on the target business service identifier, deduplication duration information, and partition creation parameters, create the corresponding Bloom filter for the partition in the creation address of the distributed service system.

[0302] In some implementations, the creation unit is also used for:

[0303] Based on the target business service identifier, deduplication duration information, and partition creation parameters, generate multiple filter indexes corresponding to at least one deduplication time period;

[0304] For each deduplication time period, multiple filter indexes are sequentially detected in the creation address of the distributed service system to obtain the detection results.

[0305] Based on the detection results, target filter indexes that do not have corresponding Bloom filters are identified, and Bloom filters for the corresponding partitions of the target filter indexes are created in the creation address of the distributed service system.

[0306] In some implementations, the creation unit is also used for:

[0307] Determine the deduplication time range based on the deduplication duration information;

[0308] Based on the partition creation parameters, determine the identifiers of multiple partitions to be created corresponding to a deduplication time period;

[0309] For each deduplication period within the deduplication time range, multiple filter indexes are constructed based on the target business service identifier and the identifier of each partition to be created.

[0310] In some embodiments, the acquisition unit 401 is further configured to:

[0311] Obtain the deduplication processing request input to the target interface of the target business service by deduplicating the service process;

[0312] The request parameters defined in the target interface should include at least the receiving object identifier, deduplication information, deduplication time range, and target business service identifier.

[0313] In some implementations, the target interface type includes a deduplication write interface, and the data processing device further includes a push unit, which is also used for:

[0314] When the deduplication check results of the Bloom filter corresponding to each target partition are all received and do not contain deduplication information index, determine the target Bloom filter of the target partition that matches the current time information.

[0315] Construct a deduplication information index based on the received object identifier and deduplication information;

[0316] The deduplication information index is sent to the target Bloom filter for storage, and the target content corresponding to the deduplication information is pushed to the target receiving object corresponding to the receiving object identifier.

[0317] As described above, this application embodiment can obtain a deduplication request triggered by any target business service. This deduplication request contains various parameters defined according to a unified interface. Then, the partition creation parameters of the target business service are read from the cache. These partition creation parameters can at least represent the number of partitions of the distributed Bloom filter created for the target business service within a deduplication time period. Thus, a routing partition identifier is generated by combining the receiving object identifier. Then, the routing partition identifier and the deduplication time range are used as filtering indexes to select target partitions in the distributed service system. Each target partition corresponds to a routing partition identifier in a deduplication time period within the deduplication time range. The distributed service system has multiple partitions and Bloom filters corresponding to each partition pre-created for the target business service. Finally, the receiving object identifier and deduplication information are sent to the Bloom filter corresponding to each target partition to request the Bloom filter corresponding to the target partition to perform deduplication checks. Therefore, compared to existing technologies that use database constraints, hash tables, or third-party services to perform deduplication checks, which cannot meet the deduplication needs of a wide range of business services, this application pre-creates corresponding distributed Bloom filters based on partition creation parameters for each business service. Each Bloom filter corresponds to a partition, and each partition is divided according to the deduplication period and partition identifier to meet the deduplication detection needs of different business services. Furthermore, each business service can carry various parameters defined by a unified interface in its request to request the Bloom filter of the target partition corresponding to each business service to perform deduplication detection. In this way, unified management and concurrent processing of deduplication processing for each business service are achieved, improving the efficiency and accuracy of deduplication processing for each business service.

[0318] The specific implementation of each of the above units can be found in the previous embodiments, and will not be repeated here.

[0319] See Figure 15 , Figure 15 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. It includes a structural block of the portion of the terminal 110 that implements the embodiment of this application. The terminal can be... Figure 1 The terminal 110 shown specifically includes: a radio frequency (RF) circuit 510, a memory 515, an input unit 520, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590, among other components. Those skilled in the art will understand that... Figure 15 The terminal 110 structure shown does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0320] The RF circuit 510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 580; in addition, it transmits uplink data to the base station.

[0321] The memory 515 can be used to store software programs and modules. The processor 580 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the memory 515.

[0322] The input unit 520 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 520 may include a touch panel 531 and other input devices 532.

[0323] The display unit 540 can be used to display input or provided information, as well as various menus of the terminal. The display unit 540 may include a display panel 541.

[0324] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface.

[0325] In this embodiment, the processor 580 included in the terminal 110 can execute the data processing method of the previous embodiment.

[0326] See Figure 16 , Figure 16 This is a schematic diagram of the server structure provided in an embodiment of this application, which includes a structural block of the server 120 implementing this embodiment. The server can be... Figure 1 The server 120 shown can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 622 (e.g., one or more processors) and memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) for storing application programs 642 or data 644. The memory 632 and storage media 630 may be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server 600. Furthermore, the CPU 622 may be configured to communicate with the storage media 630 and execute the series of instruction operations in the storage media 630 on the server 600.

[0327] Server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input / output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0328] The central processing unit 622 in the server 600 can be used to execute the data processing method of the embodiments of this application.

[0329] This application also provides a computer-readable storage medium for storing a computer program for executing the data processing methods of the foregoing embodiments.

[0330] This application also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the data processing method described above.

[0331] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0332] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0333] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0334] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0335] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0336] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0337] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0338] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0339] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0340] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the deduplication request triggered by the target business service. The deduplication request carries at least the receiving object identifier, the deduplication time range, the deduplication information, and the target business service identifier. Based on the target service identifier, determine the partition creation parameters of the distributed Bloom filter for the target service from the pre-cached target deduplication configuration information; The corresponding routing partition identifier is determined based on the receiving object identifier and the partition creation parameters, and at least one target partition corresponding to the deduplication time range is selected in the distributed service system based on the target business service identifier and the routing partition identifier. The distributed service system includes pre-created partitions for deduplication configuration information of different business services, and a Bloom filter associated with each partition. The received object identifier and the deduplication information are sent to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the received object identifier and the deduplication information.

2. The method according to claim 1, characterized in that, The step of selecting at least one target partition corresponding to the deduplication time range in the distributed service system based on the target service identifier and the routing partition identifier includes: Based on the target business service identifier, determine the partition pre-created for the target business service in the distributed service system; Query multiple candidate partitions corresponding to the routing partition identifier from the pre-created partitions; From the plurality of candidate partitions, determine at least one target partition corresponding to at least one target deduplication time period included in the deduplication time range.

3. The method according to claim 2, characterized in that, The step of determining at least one target partition corresponding to at least one target deduplication time period included in the deduplication time range from the plurality of candidate partitions includes: Determine multiple target deduplication time periods corresponding to the deduplication time range; From the multiple candidate partitions, select multiple target partitions corresponding to the multiple target deduplication time periods.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the corresponding routing partition identifier based on the receiving object identifier and the partition creation parameters includes: Based on the partition creation parameters, determine the number of partitions created by the distributed Bloom filter of the target business service per unit time period; A hash calculation is performed on the identifier of the receiving object to obtain a candidate hash value; The candidate hash value is moduloed according to the number of partitions created to obtain the routing partition identifier.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the partition creation parameters of the distributed Bloom filter for the target business service from the pre-cached target deduplication configuration information based on the target business service identifier includes: By performing a deduplication check service process, the pre-cached target deduplication configuration information corresponding to the target service identifier is retrieved from the cache space; Based on the target deduplication configuration information, determine the partition creation parameters when creating a distributed Bloom filter for the target business service.

6. The method according to claim 5, characterized in that, Before determining the partition creation parameters of the distributed Bloom filter for the target business service from the pre-cached target deduplication configuration information based on the target business service identifier, the method further includes: The deduplication check service process loads the target deduplication configuration information corresponding to the target business service identifier from the deduplication configuration information public terminal. The deduplication configuration information public terminal includes at least one business service for deduplication configuration information input for deduplication checks; The loaded target deduplication configuration information is cached in the cache space.

7. The method according to claim 6, characterized in that, Before selecting at least one target partition corresponding to the deduplication time range pre-created in the distributed service system based on the routing partition identifier, the method further includes: Load the target deduplication configuration information corresponding to the target service identifier from the public terminal of the deduplication configuration information; Based on the target deduplication configuration information, determine the creation address, deduplication duration, and partition creation parameters of the distributed Bloom filter; Based on the target business service identifier, the deduplication duration information, and the partition creation parameters, a Bloom filter corresponding to the partition is created in the creation address of the distributed service system.

8. The method according to claim 7, characterized in that, The step of creating a Bloom filter corresponding to the partition in the creation address of the distributed service system according to the target business service identifier, the deduplication duration information, and the partition creation parameters includes: Based on the target business service identifier, the deduplication duration information, and the partition creation parameters, generate multiple filter indexes corresponding to at least one deduplication time period; For each deduplication time period, multiple filter indexes are sequentially detected in the creation address of the distributed service system to obtain the detection results. Based on the detection results, a target filter index that has not been created with a corresponding Bloom filter is identified, and a Bloom filter for the partition corresponding to the target filter index is created in the creation address of the distributed service system.

9. The method according to claim 8, characterized in that, The step of generating multiple filter indexes corresponding to at least one deduplication time period based on the target service identifier, the deduplication duration information, and the partition creation parameters includes: The deduplication time range is determined based on the deduplication duration information; Based on the partition creation parameters, determine multiple partition identifiers to be created corresponding to a deduplication time period; For each deduplication period within the deduplication period range, multiple filter indexes are constructed based on the target business service identifier and the identifier of each partition to be created.

10. The method according to any one of claims 1 to 9, characterized in that, The process of obtaining the deduplication request triggered by the target business service includes: Obtain the deduplication processing request input to the target interface of the target business service by deduplicating the service process; The request parameters defined by the target interface include at least the receiving object identifier, deduplication information, deduplication time range, and target business service identifier.

11. The method according to claim 10, characterized in that, The target interface type includes a deduplication write interface. After sending the receiving object identifier and the deduplication information to the Bloom filter corresponding to each target partition, the method further includes: When the deduplication check results of the Bloom filter corresponding to each target partition are all received and do not contain deduplication information index, determine the target Bloom filter of the target partition that matches the current time information. Construct a deduplication information index based on the received object identifier and the deduplication information; The deduplication information index is sent to the target Bloom filter for storage, and the target content corresponding to the deduplication information is pushed to the target receiving object corresponding to the receiving object identifier.

12. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire a deduplication request triggered by the target business service. The deduplication request carries at least the receiving object identifier, the deduplication time range, the deduplication information, and the target business service identifier. The determining unit is configured to determine the partition creation parameters of the distributed Bloom filter for the target business service from the pre-cached target deduplication configuration information based on the target business service identifier. The selection unit is used to determine the corresponding routing partition identifier based on the receiving object identifier and the partition creation parameters, and to select at least one target partition corresponding to the deduplication time range in the distributed service system based on the target service identifier and the routing partition identifier. The distributed service system includes pre-created partitions for deduplication configuration information of different business services, and a Bloom filter associated with each partition. The sending unit is configured to send the receiving object identifier and the deduplication information to the Bloom filter corresponding to each target partition, so that the Bloom filter associated with each target partition performs deduplication detection based on the receiving object identifier and the deduplication information.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the data processing method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 11.

Images