Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, relates to the technical field of computers, can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic and auxiliary driving, and comprises the following steps: obtaining a target object table and a service table comprising at least one data partition; the target object table is used for storing object information of target objects hitting the same service category, and a single record of the service table is an object bitmap corresponding to the same service index value in one data partition; index disassembling and recombining are conducted on the service table with the data partition as the unit, a recombined service table is obtained, a single record of the recombined service table is a recombined object bitmap corresponding to the same disassembling index value in one data partition, and the disassembling index value is a disassembling item of the corresponding service index value; and performing index operation of the target object corresponding to the target object table based on the recombined service table to obtain an index operation result.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and storage medium. Background Art

[0002] With the development of network technology, business data analysis has become an important foundation for business development and promotion, such as the index statistics of the operation data of the business objects of a certain category of business. In the current business data storage scheme, the objects of a certain business category are usually stored in a table A, and its general structure is (uin, group), where uin is the unique identifier of the object, and group represents the identifier of a certain business category; and the index value of a single object is stored in another table B, and its general structure is (uin, val), where val is the index value of the object; in order to perform index statistics on the objects of this business category, it is necessary to join table A and table B with uin as the granularity, and then combine statistical methods to perform index calculations. However, the number of objects in many scenarios is huge, the calculation amount of the above table join operation is extremely large, it takes too long, and the index data processing performance is poor. Summary of the invention

[0003] The present application provides a data processing method, apparatus, device and storage medium, which can significantly improve the timeliness and reliability of data processing.

[0004] In one aspect, the present application provides a data processing method, the method comprising:

[0005] Acquire a target object table and a business table including at least one data partition, wherein the target object table is used to store object information of target objects that hit the same business category, and the business table is used to store object information corresponding to associated business indicators of the business category based on a bitmap, and a single record of the business table is an object bitmap corresponding to the same business indicator value in one data partition;

[0006] The business table is disassembled and reorganized based on the index in units of data partitions to obtain a reorganized business table, wherein a single record of the reorganized business table is a reorganized object bitmap corresponding to the same disassembly index value in a data partition, and the disassembly index value is a disassembly item of the corresponding business index value;

[0007] Based on the reorganized business table, an index calculation of the target object corresponding to the target object table is performed to obtain an index calculation result.

[0008] Another aspect provides a data processing device, the device comprising:

[0009] Acquisition module: used to acquire a target object table and a business table including at least one data partition, wherein the target object table is used to store object information of target objects that hit the same business category, and the business table is used to store object information corresponding to associated business indicators of the business category based on a bitmap, and a single record of the business table is an object bitmap corresponding to the same business indicator value in one data partition;

[0010] Reorganization module: used to disassemble and reorganize the indicators of the business table in units of data partitions to obtain a reorganized business table, wherein a single record of the reorganized business table is a reorganization object bitmap corresponding to the same disassembly indicator value in a data partition, and the disassembly indicator value is a disassembly item of the corresponding business indicator value;

[0011] Index calculation module: used to perform index calculation of the target object corresponding to the target object table based on the reorganized business table to obtain the index calculation result.

[0012] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method as described above.

[0013] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement the data processing method as described above.

[0014] On the other hand, a server is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method as described above.

[0015] On the other hand, a terminal is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method as described above.

[0016] On the other hand, a computer program product or a computer program is provided, which comprises computer instructions, and when the computer instructions are executed by a processor, the data processing method as described above is implemented.

[0017] The data processing method, apparatus, device, storage medium, server, terminal, computer program and computer program product provided by this application have the following technical effects:

[0018] The technical solution of the present application obtains a target object table and a business table including at least one data partition, wherein the target object table is used to store object information of target objects that hit the same business category, and the business table is used to store object information corresponding to associated business indicators of the business category based on a bitmap, and a single record of the business table is an object bitmap corresponding to the same business indicator value in a data partition; object information is stored in the form of a bitmap to aggregate objects with the same attributes in the same bitmap, and the join operation of the table is converted into an AND operation between bitmaps, thereby reducing the computational complexity and thus reducing the computational time. In addition, the business table is disassembled and reorganized based on the indicators of the data partition to obtain a reorganized business table, wherein a single record of the reorganized business table is a reorganized object bitmap corresponding to the same disassembled indicator value in a data partition, and the disassembled indicator value is a disassembled item of the corresponding business indicator value; the indicator operation of the target object corresponding to the target object table is performed based on the reorganized business table to obtain the indicator operation result. In this way, the bitmap data can be further merged through indicator decomposition, the number of records in the business table can be compressed to a constant range, the IO time for reading the business table and the number of bitmap operations in the indicator calculation process can be reduced, the calculation amount and time consumption can be further optimized, and the indicator data processing performance can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0021] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0022] Figure 3 is a flowchart of another data processing method provided in an embodiment of the present application;

[0023] Figure 4 This is an example diagram of generating a reorganized business table provided in an embodiment of the present application;

[0024] Figure 5 It is a calculation time-consuming curve diagram of a group control experiment provided in an embodiment of the present application;

[0025] Figure 6 This is an example diagram of data partition aggregation provided by an embodiment of the present application;

[0026] Figure 7It is a schematic diagram of a data processing device provided in an embodiment of the present application;

[0027] Figure 8 It is a hardware structure block diagram of an electronic device of a data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server comprising a series of steps or submodules is not necessarily limited to those steps or submodules clearly listed, but may include other steps or submodules that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0031] A bitmap, also called a bitmap, uses one bit of binary to indicate the usage of a disk block. When its value is "0", it indicates that the corresponding disk block is free; when it is "1", it indicates that it has been allocated. Some systems use 0 as a mark that the disk block has been allocated, and "1" as a free mark (they are essentially the same, both use one bit of two states to mark the free and allocated conditions). All disk blocks on the disk have a binary bit corresponding to them. In this way, the bits corresponding to all disk blocks constitute a set called a bitmap.

[0032] See also Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, such as Figure 1As shown, the application environment may include at least a terminal 01 and a server 02. In practical applications, the terminal 01 and the server 02 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0033] The server 02 in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0034] Specifically, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data computing, storage, processing, and sharing. Cloud technology can be applied to various fields, such as medical cloud, cloud Internet of Things, cloud security, cloud education, cloud conferencing, artificial intelligence cloud services, cloud applications, cloud calls, and cloud social networking. Cloud technology is based on the cloud computing business model application. It distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space, and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a basic capability provider for cloud computing, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service)) platform will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0035] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer, or SaaS can be directly deployed on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0036] Specifically, the server 02 mentioned above may include a physical device, which may specifically include a network communication submodule, a processor, a memory, etc., and may also include software running in the physical device, which may specifically include an application program, etc.

[0037] Specifically, terminal 01 may include physical devices such as smart phones, desktop computers, tablet computers, laptops, digital assistants, augmented reality (AR) / virtual reality (VR) devices, intelligent voice interaction devices, smart home appliances, smart wearable devices, and vehicle-mounted terminal devices, and may also include software running in physical devices, such as applications.

[0038] In an embodiment of the present application, terminal 01 can submit object-related operations on the business, such as clicks, comments, etc., and server 02 can generate business indicator data in response to the above operations and store them as corresponding target object tables and business tables; server 02 can also respond to indicator data processing instructions to perform indicator decomposition and reorganization of the business table and corresponding indicator operations based on the business table and the target object table to obtain indicator operation results.

[0039] Furthermore, it is understandable that Figure 1 What is shown is merely an application environment of a data processing method, and the application environment may include more or fewer nodes, and the present application does not impose any limitation thereto.

[0040] The application environment involved in the embodiment of the present application, or the terminal 01 and the server 02 in the application environment, etc., can be a distributed system formed by connecting a client and multiple nodes (any form of computing devices in the access network, such as a server, a user terminal) through network communication. The distributed system can be a blockchain system, which can provide the above-mentioned data processing services and data storage services.

[0041] The following is an introduction to a data processing method of the present application based on the above application environment. The present application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Please refer to Figure 2 , Figure 2 It is a flowchart of a data processing method provided by an embodiment of the present application. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, as Figure 2As shown, the method may include the following steps S201-S205:

[0042] S201: Acquire a target object table and a business table including at least one data partition.

[0043] Specifically, the target object table is used to store the object information of the target object that hits the same business category, that is, to store the correspondence between the business category and the object information. The object information can be stored based on a bitmap, and a single record is an object bitmap generated by the business category information and its corresponding object information of the target object that hits the business category. Exemplarily, the business category can specifically refer to a certain business strategy. The object of this application refers to an entity that has the ability to operate the business, such as an account number or a terminal device number, etc. A single business operation can include multiple business categories. It can be understood that a single object can hit multiple business categories at the same time. In existing business scenarios, it is usually necessary to perform separate indicator data processing for the same business category. In one embodiment, the object information is the object unique identifier, and the business category information is the category unique identifier. The structure of the target object table can be (group, bm), where group is the category unique identifier, and bm is the object bitmap (bitmap) formed by the object information that hits the business category.

[0044] Specifically, the business table is used to store object information corresponding to the associated business indicators of the business category based on the bitmap. A single record of the business table is an object bitmap corresponding to the same business indicator value in a data partition, that is, the correspondence between the business indicator value and the object information is stored, and the object information corresponding to the objects with the same business indicator value generates an object bitmap in a record. The structure of the business table is (ds, val, bm), where ds refers to the data partition, val represents the business indicator value, and bm represents the object bitmap aggregated from the object information with the business indicator value val. When processing and analyzing big data, compared with joining two tables based on object information as the granularity, and then summing the val of each object to obtain the index calculation result, the join operation of the two tables is converted into an AND operation of two bitmaps. For example, the logic of index addition calculation is converted into calculating the value of sum(val*And(bm_d,bm_c)), where bm_c is the object bitmap in the target object table C, bm_d is the object bitmap with the index value val in the business table D, And represents the AND operation of two object bitmaps, and sum represents the sum. This method can significantly reduce the calculation complexity.

[0045] Specifically, the business indicator may be set based on business requirements. For example, the business indicator may be the online time, click-through rate, click volume, or number of comments of an object associated with the business category.

[0046] Specifically, data partitioning refers to the partitioning formed by storing business indicator data based on a preset partitioning strategy. The preset partitioning strategy can be, for example, partitioning based on unit time length, or partitioning based on region, etc. Exemplarily, the unit time length (one day) is used as the partition granularity, and each data partition stores indicator data of unit time length, such as a data partition storing business indicator records on the day of 20230901. It can be understood that based on actual business needs, subsequent indicator operations can be performed on the data of a single data partition or on the data of two or more data partitions. Exemplarily, the structure of a specified data partition business table D is (ds, val, bm), and a record of the business table D is (ds i ,val j ,bm valj ), ds i refers to data partition i, val j refers to the business indicator value of the jth record, bm valj Refers to the object bitmap aggregated from the object information of the jth record. In an example, a simple business table D can be [(20230901,3,bm3),(20230901,7,bm7)], that is, a business table includes a 20230901 partition and two records, whose business indicator values ​​are 3 and 7 respectively. It can be understood that the business table aggregates objects based on the business indicator values, and the object bitmaps corresponding to different business indicator values ​​are mutually exclusive, that is, there is no overlapping object information.

[0047] S203: Decomposing and reorganizing the business table based on the data partition as the index, to obtain a reorganized business table. A single record of the reorganized business table is a reorganized object bitmap corresponding to the same decomposition index value in a data partition. The decomposition index value is a decomposition item of the corresponding business index value.

[0048] It is understandable that as the business traffic increases, the cardinality of business indicator values ​​may be very large. For example, for indicators of the reading time type, the reading time of the object may be 1 minute, 2 minutes, 5 minutes, and any other time values. The number of records in the corresponding business table also increases accordingly, and the complexity of business table reading and inter-table bitmap aggregation calculation also rises linearly. For each business indicator value, an AND operation between the object bitmap of the business indicator value and the object bitmap corresponding to the business category is required, which greatly affects the data processing performance. By disassembling and reorganizing the indicators of the business table, taking the disassembly items as the record reorganization standard, and re-aggregating the object bitmaps of the same disassembly items, the number of records in the business table can be compressed to a constant range, the number of records can be reduced, the IO time of reading the business table and the number of bitmap operations in the indicator calculation process can be reduced, the amount of calculation and time consumption can be further optimized, and the indicator data processing performance can be significantly improved. The above-mentioned indicator disassembly and reorganization refers to disassembling a single business indicator value into the sum of at least one disassembled indicator value, and aggregating the object bitmaps of the same disassembled indicator value in each disassembled indicator value after each business indicator value is disassembled in the same data partition. It can be understood that when the decomposition is performed into one decomposition item, the decomposition index value is consistent with the original business index value. After the same business index value is decompositioned and before reorganization, the object bitmap corresponding to each decomposition item is consistent with the original business index value. Figure 3 , S203 may specifically include S301-S303:

[0049] S301: Decomposing the business indicator value of each record in the business table based on base conversion to obtain at least one decomposed indicator value corresponding to each business indicator value.

[0050] Specifically, the business indicator value of each row in the business table is disassembled into corresponding disassembled items to obtain at least one disassembled indicator value. Obviously, the disassembled items of the same business indicator value are different from each other, and the two disassembled item sets generated by the two business indicator values ​​may have the same disassembled indicator value as the subsequent items to be merged. The numerical disassembly of the above-mentioned base conversion refers to disassembling each business indicator value in the business table into the sum of numbers with n as a power to obtain at least one disassembled indicator value with n as a power, and n is the cardinality of the target base corresponding to the base conversion. In this way, it is ensured that the disassembled items of the same business indicator value are different from each other and different business indicator values ​​are disassembled into repeated values ​​in a low base, so as to achieve the compression of the numerical range of the indicator value in the record and greatly reduce the number of records. Usually, the base used by the business indicator value is decimal, and the target base is the base used in the disassembly and reorganization. The base of the target base is smaller than the base of the base of the business indicator value, such as n is 2 or 3.

[0051] For example, the target system may be binary, that is, each service indicator value is decomposed into the sum of numbers with 2 as a power, and at least one decomposed indicator value with 2 as a power is obtained. For example, the decomposed indicator value of 1 is 2.0 =1, the disassembly index value of 2 is 2 1 =1, the disassembly index value of 3 is 2 0 +2 1 , that is, 1 and 2. The record (20230901,3,bm3) in the above business table D[(20230901,3,bm3),(20230901,7,bm7)] is decomposed into (20230901,1,bm3) and (20230901,2,bm3), and the record (20230901,7,bm7) is decomposed into (20230901,1,bm7), (20230901,2,bm7) and (20230901,4,bm7).

[0052] S303: for each data partition in the business table, aggregate the object bitmaps corresponding to the same disassembly index value to obtain a reorganized object bitmap corresponding to each disassembly index value, and generate a reorganized business table.

[0053] Specifically, in a single data partition, the same object information in the business table has only one business indicator value, that is, the object information in the object bitmaps of different business indicator values ​​are mutually exclusive, that is, the AND operation of bm in any two records in the business table will return an empty bm. Object bitmap aggregation in S303 refers to the OR operation of the object bitmap, that is, the union of each object bitmap is used as the reorganized object bitmap. The indicator decomposition by remapping the numerical range through base conversion can achieve efficient decomposition of indicators and compression of the number of records, thereby reducing the subsequent indicator calculation amount.

[0054] In some embodiments, S303 may specifically include S3031-S3033:

[0055] S3031: for each data partition in the business table, determine the object bitmap corresponding to each disassembly index value;

[0056] S3032: for each disassembly index value, perform an OR operation on each object bitmap corresponding to the disassembly index value to obtain a reorganized object bitmap;

[0057] S3033: Generate a reorganization business table by taking each disassembly index value and the reorganization object bitmap corresponding to the disassembly index value as a single record.

[0058] Specifically, after each business indicator value in the business table is disassembled, the same disassembled indicator value is selected from each disassembled item generated by the business table, and the object bitmap corresponding to each disassembled indicator value is a bitmap set formed by the object bitmaps corresponding to each business indicator value that generates the disassembled indicator value. Further, for each bitmap set corresponding to the disassembled indicator value, the union of the object bitmaps in the bitmap set is taken to form a reorganized object bitmap, and then each record formed by the disassembled indicator value and the reorganized object bitmap is reorganized, and the disassembled indicator values ​​of different records in the reorganized business table are different. The business table is reorganized by simple numerical disassembly and bitmap or operation, and the calculation is simple and efficient.

[0059] Taking the aforementioned business table D[(20230901,3,bm3),(20230901,7,bm7)] as an example, in the 20230901 data partition, the object bitmap corresponding to the disassembly index value 1 is [bm3,bm7], the object bitmap corresponding to the disassembly index value 2 is [bm3,bm7], and the object bitmap corresponding to the disassembly index value 4 is bm7; then, through the OR operation, the records in the reorganized business table are (20230901,1,bm3∪bm7), (20230901,3,bm3∪bm7) and (20230901,4,bm7) respectively.

[0060] It can be understood that in binary disassembly and reorganization, the structure of the reorganized business table E is also (ds, val', bm'), where the value range of val' is fixed to the power of 2. For example, a reorganized business table E can be [(20230901,1,bm1'),(20230901,2,bm2'),(20230901,4,bm4')...(20230901,1024,bm 1024 ')], it is obvious that the number of records in the reorganized business table E is compressed into a constant range.

[0061] S205: Perform index calculation on the target object corresponding to the target object table based on the reorganized business table to obtain the index calculation result.

[0062] It can be understood that a single record in the target object table is a business category and its corresponding object bitmap. In the process of performing index calculations for a specified business category, the object bitmap of the specified business category needs to be ANDed with the object bitmap in the reorganized business table to determine the overlapping object information, thereby realizing the index calculation of the associated objects of the specified business category. Index calculations for the same business category involve index calculation scenarios for a single data partition, as well as index calculation scenarios for two or more data partitions. Accordingly, in the case of performing index calculations based on records of the same data partition in the reorganized business table, that is, index calculation scenarios for a single data partition, S205 may include the following steps S401-S407:

[0063] S401: Based on the target object table, determine the object intersection between the reorganization object bitmap of each record in the same data partition in the reorganization business table and the target object hitting the same business category, and obtain the business category object set corresponding to each disassembly index value in the data partition.

[0064] Specifically, the reorganized object bitmap of each record in a single data partition for which index calculation is required is ORed with the object bitmap of the specified business category to determine overlapping object information, i.e., object information that belongs to the specified business category and has the disassembly index value in the record, to form a business category object set, the storage data format of which can still be a bitmap.

[0065] S403: Obtain a preset algorithm for the indicator calculation corresponding to the business category.

[0066] Specifically, the embodiments of the present application can be applied to an indicator calculation scenario with a requirement to add up the values ​​of multiple object indicators, and the preset algorithm is determined based on the specific indicator calculation requirement. For example, if the indicator calculation scenario is to calculate the average reading time of the target object that hits a specified business category, the preset algorithm is Among them, T average is the average reading time per person (the result of the indicator calculation), T total The total reading time of the target objects in the specified business category, m total is the total number of target objects that hit the specified business category, k-1 is the number of records in the data partition that needs to perform index calculation in the reorganized business table, and al j ′ is the reading time (disassembly index value) of the jth record in the data partition, m j The number of elements in the business category object set corresponding to the disassembly index value of the j-th record.

[0067] S405: Perform indicator statistics calculation based on the disassembly indicator value and the number of elements in the business category object set to obtain statistical indicator data.

[0068] S407: Performing index calculation using the statistical index data as input of a preset algorithm to obtain an index calculation result.

[0069] Specifically, for the data partition, after determining the business category object set that each disassembled index value hits on the specified business category, determine the number of elements corresponding to each business category object set, that is, the number of objects, to achieve index statistics. The index statistics here can be statistical methods such as summing up the index values. For example, to calculate the total index value of the specified business category, multiply the disassembled index value of each record by the number of elements of the business category object set corresponding to it, and obtain the sum of the single indicators corresponding to the disassembled index value, and then add up the sum of the single indicators of all records in the specified data partition, and determine the obtained sum of the index values ​​as statistical index data. It can be understood that the statistical index data is not limited to the sum of the index values, and can be other statistical results based on scenario requirements. In this way, the object intersection operation is performed through the bitmap to determine the object information of the business category, and then the disassembled index value and the number of object information of each record are combined to perform efficient index operations, reduce calculation time, and improve calculation performance.

[0070] It should be noted that in various indicator calculation scenarios, the sum of indicator values ​​is a common indicator statistical calculation method, and the indicator value of a single record can be used to perform cumulative calculations on the number of objects. It is highly compatible with the calculation of the reorganized business table and the calculation method is simple. Therefore, in various indicator calculation scenarios, the elements in the preset algorithm can be decomposed into the sum of indicator values, and then substituted into the preset algorithm for data calculation to reduce the calculation complexity. For example, in the aforementioned average reading time scenario, its numerator is decomposed into the sum of indicator values.

[0071] It can be understood that before generating the reorganized business table E, the number of records in each data partition in the business table D is relatively large. Taking the calculation of the sum of index values ​​as an example, the index operation between the business table D and the target object table C can be represented as sum(val_d*And(bm_d,bm_c)). After conversion to the reorganized business table E, the index operation between it and the target object table C can be represented as sum(val_e*And(bm_e,bm_c)), where bm_e is the object bitmap in the reorganized business table E. Obviously, the results before and after the conversion are the same.

[0072] In one example, combining Figure 4, the business table D is [(1,bm1),(2,bm2),(3,bm3),(5,bm5),(7,bm7)], which has only one data partition. After step S301, the business index values ​​of the business table D are decomposed into [(1,bm1),(2,bm2),(1+2,bm3),(1+4,bm5),(1+2+4,bm7)], and the corresponding records are decomposed into [(1,bm1),(2,bm2), (1,bm3),(2,bm3),(1,bm5),(4,bm5),(1,bm7),(2,bm7),(4,bm7)], since bm1, bm2, bm5, bm7 are mutually exclusive, the object bitmaps with the same disassembly index value are aggregated to obtain a reorganized business table E, specifically [(1,bm1Ubm3Ubm5Ubm7),(2,bm2Ubm7),(4,bm5Ubm7)]. Among them, ∪ represents the OR operation of two bitmaps, ∩ represents the AND operation of two bitmaps, the calculation expression based on the sum of the indicator values ​​of business table D and target object table C is 1*(bm1∩bm_c)+2*(bm2∩bm_c)+3*(bm3∩bm_c)+5*(bm5∩bm_c)+7*(bm7∩bm_c), and the calculation expression based on the sum of the indicator values ​​of the reorganized business table E and the target object table C is 1*((bm1Ubm3Ubm5Ubm7)∩bm_c)+2*((bm2Ubm3)∩bm_c)+4*(bm5Ubm7)∩bm_c), and it is obvious that the two are equal.

[0073] In combination with the above-mentioned data processing method, an efficient indicator calculation method based on the decomposition and reorganization of indicator values ​​is carried out in the experimental platform provided by this application, which achieves a substantial compression of the number of business table records and speeds up the calculation of indicators. Under the same experimental conditions, the business table and the target object table are directly used for indicator calculation as the control group, and the reorganized business table and the target object table are used for indicator calculation as the experimental group. The number of records in the business table with a single data partition is 110165893, and the number of records in the reorganized business table is 809623. The compression ratio is 136, which greatly reduces the IO time for reading data. The calculation time for the experimental group (calculation time after disassembly) and the control group (calculation time without disassembly) to perform indicator addition calculation is shown in the figure. Figure 5 As shown in the figure, in 50 consecutive calculations, the total time consumed for the index calculation of the control group is 124601ms, and the total time consumed for the index calculation of the experimental group is 25975ms, which is 80% less. Understandably, in the case of multiple data partitions, the record compression of the business table is more significant.

[0074] As mentioned above, there are also cases where index operations are performed based on the records of at least two data partitions in the reorganized business table. In some cases, the record group in each data partition is partitioned and joined with the target object table C, and the index operations of the single partitions are performed separately in the aforementioned S401-S407 manner, and then the results of each data partition are summed to obtain the index operation result. Obviously, the partition complexity of this method is the number V of data partitions required for index operations. In other cases, based on the carry principle of the adder, the records of at least two data partitions that need to be indexed are aggregated to reduce the number of data partitions, thereby reducing the partition complexity of the calculation. Accordingly, S205 may include the following steps S501-S503:

[0075] S501: Taking the same disassembly index value as an aggregation standard, based on the adder method, the records of at least two data partitions are progressively aggregated in a partition table layer by layer to obtain a target business table corresponding to the at least two data partitions.

[0076] Specifically, a single record of the target business table is an aggregated object bitmap corresponding to the index value obtained after aggregation. The adder method adopted above may include but is not limited to the carry method of the full adder; the progressive partition table aggregation layer by layer may be to carry out the record carry operation of the same disassembly index value with two data partitions as a group, so as to perform the disassembly index value and the reorganization object bitmap of the record group of at least two data partitions based on the disassembly index value Iteratively carry operation layer by layer, until the disassembly index values ​​of each record after aggregation are different from each other, and the target business table corresponding to at least two data partitions for the required index operation operation is obtained.

[0077] In some embodiments, S501 may specifically include S5011-S5013:

[0078] S5011: Combining at least two data partitions in sequence, with two data partitions as a group, to obtain at least one data partition group.

[0079] Specifically, the reorganized business table is (ds, val', bm'), ds represents the data partition, and a single record is (ds i ,val j ',bm valj '),ds i Represents data partition i, val j 'For ds i The disassembly index value of the jth record in bm valj 'For ds i The reorganization object bitmap of the jth record in the example, the reorganization business table of one can be [(20230901,1,bm 11 '),(20230902,1,bm 12'),(20230903,2,bm 23 ')], which includes three data partitions (20230901, 20230902 and 20230903), and their corresponding three records ((20230901,1,bm 11 '),(20230902,1,bm 12 ') and (20230903,2,bm 23 ')).

[0080] Specifically, two consecutive adjacent data partitions are taken as a data partition group, and the data partitions in different data partition groups do not overlap with each other. The data partitions in the reorganized business table can be arranged in time sequence. In the above example, data partitions 20230901 and 20230902 are grouped together, and data partition 20230903 is grouped as a bye and directly enters the next layer of carry aggregation.

[0081] Specifically, a column of values ​​h is added to each record in the reorganized business table to represent the number of historical aggregations. The table structure is converted to an added column business table F(ds,h,val',bm'). The number of historical aggregations is used to indicate the number of partition additions experienced to generate the current mapped data partition and the corresponding aggregation record group. In full adder calculation, h can be the number of binary carry layers. The above example is converted to [(20230901,0,1,bm 11 '),(20230902,0,1,bm 12 '),(20230903,0,2,bm 23 ')], it can be understood that in the initial state, record aggregation has not been performed between the data partitions, so the number of historical aggregations h is 0. After a data grouping and addition operation, h increases to 1.

[0082] S5012: Use the two record groups corresponding to each data partition group as input for addition operation, and perform progressive carry aggregation on the records with the same disassembly index value in the two record groups until the index values ​​of the aggregated records are different, and then map each data group into a single data partition and aggregate the corresponding two record groups into an aggregated record group.

[0083] It is understandable that there may be overlapping object information between object bitmaps corresponding to the same disassembly index value in different data partitions, such as the continuous reading time of the same object in two consecutive days may be 1 minute. Accordingly, for each data partition group, the XOR set and intersection set between the reorganized object bitmaps of two records with the same disassembly index value in the two record groups are determined. The object information in the XOR set is the non-overlapping object information in the two records with the same disassembly index value, and the object information in the intersection set is the overlapping object information in the two records with the same disassembly index value.

[0084] Furthermore, index values ​​are generated for the XOR set and the intersection respectively to merge the two record groups into an updated record group, and the corresponding data partition group is mapped into a data partition. For each pair of records with the same disassembly index value, an XOR set and an intersection are generated to achieve aggregation of the object bitmap, and an aggregate index value for each XOR set and intersection is generated to combine the records of the two data partitions into a record in one data partition. For a single record in the updated record group, the aggregate index value corresponding to the XOR set is the original disassembly index value, and the aggregate index value corresponding to the intersection is double the original disassembly index value, that is, a carry operation is performed on the disassembly index value of the intersection. As mentioned above, the record group (20230901,0,1,bm 11 ') and record group (20230902,0,1,bm 12 '), the two reorganized object bitmaps corresponding to the disassembly index value 1 are bm 11 ' and bm 12 ', perform an addition operation on the non-overlapping parts of the two reorganized object bitmaps, and perform a carry operation on the overlapping object information parts, and perform a carry operation on the aggregation index value val of the carry operation part j ” is 2val j ', for example [(20230901,0,1,bm 11 ')] can be seen as [(20230901,0,1,bm 111 '∪bm 112 '],[(20230902,0,1,bm 12 ')] can be seen as [(20230902,0,1,bm 122 '∪bm 112 '], where bm 112 '=bm 11 '∩bm 12 ', bm 111 '=bm 11 '-bm 112 ', bm 122 '=bm 12 '-bm 112 ', bm 112 ' is the intersection, bm 111 '∪bm 122 ' is an XOR set, the two consecutive data partitions 20230901 and 20230902 carry 20230902, and the update record group of the mapped data partition is (20230902,1,{(1,bm 111 '∪bm 122 '),(2,bm 112 ')}), it is obvious that the data is obtained by summing the above two data partitions, and the original two data partitions are compressed into the data of one data partition.

[0085] Furthermore, for a data partition after mapping, for two records with the same aggregate index value in the corresponding update record group, the steps of determining the XOR set and the intersection, as well as the step of generating the index value are repeated until the aggregate index values ​​of the records in the updated reorganization business table are different, thereby obtaining an aggregate record group. In the above example, each data partition contains only one record. It can be understood that a single data partition may include multiple records, and the disassembly index value of each record is different. For example, the disassembly index values ​​of the three records in the first data partition are 1, 2, and 4, respectively, and the disassembly index values ​​of the three records in the second data partition are 1 and 2, respectively; then for the two data partitions, the reorganization object bitmap of the disassembly index value 1 is determined by the XOR set and the intersection, and the reorganization object bitmap of the disassembly index value 2 is determined by the XOR set and the intersection, and the disassembly index value 4 is left blank this time to generate the corresponding update record group (1-XOR set, 2-intersection, 2 -XOR set, 4-intersection, 4-reorganized object bitmap), so that the two data partitions are mapped to the second data partition; for the updated record group in the mapped second data partition, it is obvious that the data with the same disassembly index value needs to be further aggregated, 2-intersection and 2-XOR set are records with the same disassembly index value, 4-intersection and 4-reorganized object bitmap are records with the same index value, and the XOR set and intersection determination and index carry operation are repeated, and so on, until the disassembly index values ​​in each record are different, and the disassembly index values ​​of each record in the final aggregated record group are 1, 2, 3, 8, and 16. It can be understood that the h value increases by 1 without performing the sequential addition operation.

[0086] S5013: In the case where there are multiple groups of data partition groups, for the multiple mapped data partitions obtained, the above-mentioned data partition combination and addition operation steps are executed cyclically until the partition aggregation termination condition is met, and the target business table is generated based on the mapped data partitions obtained when the partition aggregation termination condition is met and their corresponding records.

[0087] It can be understood that in the case of multiple data partition groups, multiple data partitions are mapped after the first round, and then the mapped multiple data partitions are further grouped into two data partitions based on sorting to obtain updated data partition groups, and then the aforementioned addition operation is performed on each updated data partition group. It should be noted that the h values ​​of the two data partitions in the data partition group are the same, and this process is repeated until there are no consecutive adjacent data partitions with the same h value, and it is determined that the partition aggregation termination condition is met, and the record data of the mapped data partition is used as the target business table.

[0088] It can be understood that when at least two data partitions are odd numbers, there is a data partition that cannot be grouped. In the operation process of the above-mentioned adder, the record group of the ungrouped data partition is directly passed without aggregation calculation, and its corresponding h remains unchanged and is directly used as a data partition in the target business table.

[0089] In this way, partition integration is achieved by aggregating the indicator data within the data partitions, and the indicator integration of overlapping object information is performed in advance. When performing indicator operations, there is no need to perform separate bitmaps and operations on each original data partition and target object table, that is, to perform indicator operations from ds1 to dsn. Instead, it can be decomposed and carried into the sum of the calculation results of the [(dsi,hi)] data partitions, reducing the partition complexity to log2(V), significantly improving the computing efficiency.

[0090] S503: Perform an index calculation on the target object corresponding to the target object table based on the target business table to obtain an index calculation result.

[0091] It can be understood that the number of data partitions in the target business table is significantly reduced. For each data partition in the target business table, the aforementioned S401-S407 method is used to perform index calculations, and the calculation results of each data partition are added to obtain the index calculation results. Ideally, the target business table has only one mapped and aggregated data partition. In this way, the number of records in a single partition is reduced while the partition complexity is reduced, significantly improving the calculation efficiency.

[0092] In an example, it is necessary to calculate the indicator data from 20230901 to 20230905. In this case, the business table E[(20230901,1,bm11),(20230902,1,bm2),(20230903,1,bm3),(20230904,1,bm4),(20230905,1,bm5)] is reorganized. Figure 6 After h columns are added, a new table F (h=0 partition) is obtained. At this time, there are 5 data partitions. According to the full adder carry principle, the bm in these 5 data partitions are added, and finally the two partitions (20230904,2) and (20230905,0) are obtained. When calculating the indicator data from 20230901 to 20230905, only the indicator data in the two data partitions [(20230904,2) and (20230905,0)] in Table F need to be summed, and the IO data volume is reduced by 60%.

[0093] This application compresses the number of records in the business table D within a constant range based on the indicator numerical decomposition algorithm, reduces the IO time for reading the business table and the number of bitmap operations, and realizes the merging of zone data through the full addition and carry operation of data partitions, further reducing the partition complexity, thereby achieving efficient calculation of indicators.

[0094] The present application embodiment also provides a data processing device 600, such as Figure 7 As shown, Figure 7 A schematic structural diagram of a data processing device provided in an embodiment of the present application is shown, and the device may include the following modules.

[0095] Acquisition module 10: used to acquire a target object table and a business table including at least one data partition, wherein the target object table is used to store object information of target objects that hit the same business category, and the business table is used to store object information corresponding to associated business indicators of the business category based on a bitmap, and a single record of the business table is an object bitmap corresponding to the same business indicator value in a data partition;

[0096] Reorganization module 20: used to disassemble and reorganize the indicators of the business table in units of data partitions to obtain a reorganized business table, wherein a single record of the reorganized business table is a reorganization object bitmap corresponding to the same disassembly indicator value in a data partition, and the disassembly indicator value is a disassembly item of the corresponding business indicator value;

[0097] The index calculation module 30 is used to perform the index calculation of the target object corresponding to the target object table based on the reorganized business table to obtain the index calculation result.

[0098] In some embodiments, the recombination module 20 comprises:

[0099] Numerical decomposition submodule: used to perform numerical decomposition based on base conversion on the business indicator value of each record in the business table, and obtain at least one decomposition indicator value corresponding to each business indicator value;

[0100] Bitmap aggregation submodule: for each data partition in the business table, aggregating the object bitmaps corresponding to the same disassembly index value, obtaining the reorganized object bitmap corresponding to each disassembly index value, and generating a reorganized business table.

[0101] In some embodiments, the numerical decomposition submodule is specifically used to: decompose each business indicator value in the business table into the sum of numbers raised to the power of n, and obtain at least one decomposed indicator value raised to the power of n, where n is the base of the target base corresponding to the base conversion.

[0102] In some embodiments, the bitmap aggregation submodule includes:

[0103] A bitmap determination unit: used to determine the object bitmap corresponding to each disassembly index value for each data partition in the business table;

[0104] OR operation unit: used for performing an OR operation on each object bitmap corresponding to each disassembly index value to obtain a reorganized object bitmap;

[0105] The reorganization table generation unit is used to generate a reorganization business table by taking each disassembly index value and the reorganization object bitmap corresponding to the disassembly index value as a single record.

[0106] In some embodiments, when index calculation is performed based on records of the same data partition in the reorganized business table, the index calculation module 30 includes:

[0107] Object intersection submodule: used to determine the object intersection between the reorganized object bitmap of each record in the same data partition in the reorganized business table and the target object hitting the same business category based on the target object table, and obtain the business category object set corresponding to each disassembly index value in the data partition;

[0108] Algorithm acquisition submodule: used to obtain the preset algorithm for the indicator operation corresponding to the business category;

[0109] Statistical calculation submodule: used to perform statistical calculation of indicators based on the disassembly indicator value and the number of elements of the business category object set to obtain statistical indicator data;

[0110] Index calculation submodule: used to perform index calculation using statistical index data as input of preset algorithm to obtain index calculation results.

[0111] In some embodiments, when index calculation is performed based on records of at least two data partitions in the reorganized business table, the index calculation module 30 includes:

[0112] Partition aggregation submodule: used to perform progressive partition table aggregation on records of at least two data partitions based on the adder method with the same disassembly index value as the aggregation standard, to obtain the target business table corresponding to at least two data partitions, and a single record of the target business table is an aggregation object bitmap corresponding to the index value obtained after aggregation;

[0113] The index calculation submodule is also used to perform index calculation of the target object corresponding to the target object table based on the target business table to obtain the index calculation result.

[0114] In some embodiments, the partition aggregation submodule includes:

[0115] A combining unit: used for combining at least two data partitions in sequence, with two data partitions as a group, to obtain at least one data partition group;

[0116] Addition operation unit: used to perform addition operation with two record groups corresponding to each data partition group as input, and to perform progressive carry aggregation with records with the same disassembly index value in the two record groups until the index values ​​of the aggregated records are different, thereby mapping each data group into a single data partition and aggregating the corresponding two record groups into an aggregated record group;

[0117] Target table generation unit: used for, in the case of multiple groups of data partition groups, cyclically executing the above-mentioned steps of data partition combination and addition operation for the multiple mapped data partitions obtained until the partition aggregation termination condition is met, and generating a target business table based on the mapped data partitions obtained when the partition aggregation termination condition is met and their corresponding records.

[0118] In some embodiments, the addition unit includes:

[0119] Logical operation subunit: used for determining, for each data partition group, the exclusive OR set and the intersection between the reorganized object bitmaps of two records with the same disassembly index value in two record groups;

[0120] Index value generation subunit: used to generate index values ​​for XOR sets and intersection sets respectively, so as to merge two record groups into one update record group, and map the corresponding data partition groups into one data partition. For a single record in the update record group, the aggregate index value corresponding to the XOR set is the original disassembly index value, and the aggregate index value corresponding to the intersection set is double the original disassembly index value.

[0121] Update record group subunit: used to merge two record groups into one update record group, and map the corresponding data partition group into one data partition. For a single record in the update record group, the aggregation index value corresponding to the XOR set is the original disassembly index value, and the aggregation index value corresponding to the intersection set is double the original disassembly index value.

[0122] Aggregate record group subunit: used to repeatedly perform the steps of determining the XOR set and the intersection, and generating the index value for two records with the same aggregate index value in the corresponding update record group for a mapped data partition, until the aggregate index values ​​of the records in the updated reorganized business table are different, thereby obtaining an aggregate record group.

[0123] It should be noted that the above device embodiment and method embodiment are based on the same implementation method.

[0124] An embodiment of the present application provides a data processing device, which can be a terminal or a server, including a processor and a memory, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method provided in the above method embodiment.

[0125] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0126] The method embodiments provided in the embodiments of the present application can be executed in electronic devices such as mobile terminals, computer terminals, servers or similar computing devices. Figure 8 1 is a hardware structure block diagram of an electronic device of a data processing method provided in an embodiment of the present application. Figure 8 As shown, the electronic device 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be short-term storage or permanent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 and execute a series of instruction operations in the storage medium 920 on the electronic device 900. The electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or, one or more operating systems 921, such as Windows Server TM , Mac OS X TM , Unix TM , LinuxTM, FreeBSDTM, etc.

[0127] The input / output interface 940 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 900. In one example, the input / output interface 940 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 940 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0128] It can be understood by those skilled in the art that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer components as shown, or with Figure 8 Different configurations shown.

[0129] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to a data processing method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method provided by the above method embodiment.

[0130] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0131] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0132] It can be seen from the embodiments of the data processing method, device, equipment, server, terminal, storage medium and program product provided by the present application that the technical solution of the present application adds at least one target scheduled task corresponding to the current time slice to the corresponding queue of pending tasks through the target scheduler, wherein the target scheduler is a scheduler among multiple schedulers that successfully locks the current time slice. In this way, the scheduled tasks are prevented from being repeatedly processed through the locking mechanism, the scheduling reliability and controllability are improved, and the breakpoint resumption can be achieved in the scheduler restart scenario, avoiding re-locking and reducing the task omission rate; then, the target scheduler is controlled to update the scheduling execution information of at least one target scheduled task in the task record, wherein the task record includes the scheduling execution information of historical scheduled tasks corresponding to multiple historical time slices; the target scheduler is controlled to perform overdue detection on the historical scheduled tasks based on the scheduling execution information of the historical scheduled tasks in the task record to obtain the overdue detection result; when the overdue detection result is that there is a target overdue task, the target overdue task is added to the corresponding queue of pending tasks through the target scheduler. In this way, after the target scheduler completes the target scheduled task of the current time slice, it can simultaneously update the scheduling execution information and screen overdue tasks by obtaining task records without any additional overhead. In addition, the target scheduler can realize timely backtracking and scheduling of overdue tasks to share the scheduling task volume of other schedulers without relying on a backup server, effectively improving the timeliness and reliability of task scheduling.

[0133] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0134] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0136] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a target object table and a business table including at least one data partition, wherein the target object table is used to store object information of target objects that hit the same business category, and the business table is used to store object information corresponding to associated business indicators of the business category based on a bitmap, and a single record of the business table is an object bitmap corresponding to the same business indicator value in one data partition; The business table is disassembled and reorganized based on the index in units of data partitions to obtain a reorganized business table, wherein a single record of the reorganized business table is a reorganized object bitmap corresponding to the same disassembly index value in a data partition, and the disassembly index value is a disassembly item of the corresponding business index value; Based on the reorganized business table, an index calculation of the target object corresponding to the target object table is performed to obtain an index calculation result.

2. The method according to claim 1, characterized in that The step of disassembling and reorganizing the business table based on data partitions to obtain a reorganized business table includes: Performing numerical decomposition based on base conversion on the business indicator value of each record in the business table to obtain at least one decomposition indicator value corresponding to each business indicator value; For each data partition in the business table, the object bitmaps corresponding to the same disassembly index value are aggregated to obtain a reorganized object bitmap corresponding to each disassembly index value, and the reorganized business table is generated.

3. The method according to claim 2, characterized in that The step of performing numerical decomposition based on base conversion on the business indicator value of each record in the business table to obtain at least one decomposition indicator value corresponding to each business indicator value includes: Each business indicator value in the business table is decomposed into a sum of numbers raised to a power of n, to obtain at least one decomposed indicator value raised to a power of n, where n is the base of the target system corresponding to the system conversion.

4. The method according to claim 2, characterized in that: For each data partition in the business table, aggregating the object bitmaps corresponding to the same disassembly index value to obtain the reorganized object bitmap corresponding to each disassembly index value, and generating the reorganized business table includes: For each data partition in the business table, determine an object bitmap corresponding to each disassembly index value; For each of the disassembly index values, performing an OR operation on each object bitmap corresponding to the disassembly index value to obtain the reorganized object bitmap; The reorganization business table is generated by taking each of the disassembly index values ​​and the reorganization object bitmap corresponding to the disassembly index value as a single record.

5. The method according to any one of claims 1 to 4, characterized in that In the case where the index calculation is performed based on the records of the same data partition in the reorganized business table, the index calculation of the target object corresponding to the target object table is performed based on the reorganized business table to obtain the index calculation result, including: Based on the target object table, determine the object intersection between the reorganization object bitmap of each record in the same data partition in the reorganization business table and the target object that hits the same business category, and obtain the business category object set corresponding to each disassembly index value in the data partition; Obtaining a preset algorithm for calculating the indicator corresponding to the business category; Performing indicator statistical calculation based on the disassembly indicator value and the number of elements in the business category object set to obtain statistical indicator data; The statistical indicator data is used as the input of the preset algorithm to perform indicator calculation to obtain the indicator calculation result.

6. The method according to any one of claims 1 to 4, characterized in that In the case where the index calculation is performed based on the records of at least two data partitions in the reorganized business table, the index calculation of the target object corresponding to the target object table is performed based on the reorganized business table to obtain the index calculation result, including: Taking the same disassembly index value as the aggregation standard, based on the adder method, the records of the at least two data partitions are progressively aggregated in the partition table layer by layer to obtain the target business table corresponding to the at least two data partitions, wherein a single record of the target business table is an aggregation object bitmap corresponding to the index value obtained after aggregation; An index calculation of the target object corresponding to the target object table is performed based on the target business table to obtain the index calculation result.

7. The method according to claim 6, characterized in that The step of performing progressive partition table aggregation on the records of the at least two data partitions based on the adder method with the same disassembly index value as the aggregation standard to obtain the target business table corresponding to the at least two data partitions includes: Combining the at least two data partitions in sequence, with two data partitions as a group, to obtain at least one data partition group; Taking the two record groups corresponding to each of the data partition groups as inputs for addition operation, performing progressive carry aggregation on the records with the same disassembly index value in the two record groups until the index values ​​of the aggregated records are different, and then mapping each of the data groups into a single data partition and aggregating the corresponding two record groups into an aggregated record group; In the case where there are multiple groups of data partition groups, the above-mentioned steps of data partition combination and addition operation are executed cyclically for the multiple mapped data partitions obtained until the partition aggregation termination condition is met, and the target business table is generated based on the mapped data partitions obtained when the partition aggregation termination condition is met and their corresponding records.

8. The method according to claim 7, characterized in that The method of taking the two record groups corresponding to each of the data partition groups as inputs for addition operation, performing progressive carry aggregation on the records with the same disassembly index value in the two record groups until the index values ​​of the aggregated records are different, and then mapping each of the data groups into a single data partition and aggregating the corresponding two record groups into an aggregated record group comprises: For each of the data partition groups, determining an exclusive OR set and an intersection between the reorganization object bitmaps of two records with the same disassembly index value in the two record groups; Generate index values ​​for the XOR set and the intersection set respectively, so as to merge the two record groups into an update record group, and map the corresponding data partition group into one data partition, wherein for a single record of the update record group, the aggregate index value corresponding to the XOR set is the original disassembly index value, and the aggregate index value corresponding to the intersection set is double the original disassembly index value; The two record groups are merged into an update record group, and the corresponding data partition group is mapped into one data partition. For a single record of the update record group, the aggregation index value corresponding to the XOR set is the original disassembly index value, and the aggregation index value corresponding to the intersection set is double the original disassembly index value; For a data partition after mapping, for two records with the same aggregation index value in the corresponding update record group, the steps of determining the XOR set and the intersection, and the step of generating the index value are repeated until the aggregation index values ​​of the records in the updated reorganized business table are different, thereby obtaining the aggregate record group.

9. A data processing device, characterized in that: The device comprises: Acquisition module: used to acquire a target object table and a business table including at least one data partition, wherein the target object table is used to store object information of target objects that hit the same business category, and the business table is used to store object information corresponding to associated business indicators of the business category based on a bitmap, and a single record of the business table is an object bitmap corresponding to the same business indicator value in one data partition; Reorganization module: used to disassemble and reorganize the indicators of the business table in units of data partitions to obtain a reorganized business table, wherein a single record of the reorganized business table is a reorganization object bitmap corresponding to the same disassembly indicator value in a data partition, and the disassembly indicator value is a disassembly item of the corresponding business indicator value; Index calculation module: used to perform index calculation of the target object corresponding to the target object table based on the reorganized business table to obtain the index calculation result.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 8.

11. A computer device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 8.

Images