Data processing method and related device

Abstract

The application discloses a kind of data processing method and related equipment related to computer technology field, specifically related to a kind of data processing method and related equipment, the method includes: obtaining query request;Parsing condition limiting information, determine at least two query conditions and the Boolean logic relation between at least two query conditions corresponding to condition limiting information;At least two query conditions each query condition corresponding to query code is generated;Query code is used to carry out object query in high-efficiency compressed bitmap data table, obtain the first high-efficiency compressed bitmap of object set satisfying the corresponding query condition;According to the Boolean logic relation between at least two query conditions, the query code corresponding to each query condition in at least two query conditions is combined, and combined code is obtained;Combined code is used to carry out logical operation on the first high-efficiency compressed bitmap according to the Boolean logic relation between at least two query conditions, and obtain target high-efficiency compressed bitmap, the present scheme realizes block query, improves query efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, to a data processing method and related device. BACKGROUND

[0002] When a user needs to query information in a database system, the user needs to input query information, which defines the conditions of the objects to be queried. If the conditions defined by the query information are complex, it takes a long time to query in the database, and the query efficiency is low. SUMMARY

[0003] In view of the above problems, embodiments of the present application provide a data processing method and related device to improve the above problems.

[0004] According to an aspect of an embodiment of the present application, a data processing method is provided, including: obtaining a query request, the query request indicating query information, the query information including condition limiting information; parsing the condition limiting information to determine at least two query conditions corresponding to the condition limiting information and a Boolean logical relationship between the at least two query conditions; generating a query code corresponding to each of the at least two query conditions respectively; the query code is used to query objects in a high-efficiency compressed bitmap data table to obtain a first high-efficiency compressed bitmap of an object set satisfying the corresponding query condition, the high-efficiency compressed bitmap data table including high-efficiency compressed bitmaps of a plurality of initial object sets; the initial object set is determined by classifying objects according to attribute values; combining the query codes corresponding to each of the at least two query conditions according to the Boolean logical relationship between the at least two query conditions to obtain a combined code; the combined code is used to perform logical operations on the first high-efficiency compressed bitmap according to the Boolean logical relationship between the at least two query conditions to obtain a target high-efficiency compressed bitmap; and the target high-efficiency compressed bitmap is used to determine a query result.

[0005] According to an aspect of some embodiments of the present application, a data processing apparatus is provided, comprising: a query request obtaining module configured to obtain a query request, the query request indicating query information, the query information comprising condition limiting information; an analysis module configured to analyze the condition limiting information, determine at least two query conditions corresponding to the condition limiting information and a Boolean logic relationship between the at least two query conditions; a query code generating module configured to generate a query code corresponding to each of the at least two query conditions respectively; the query code being used to perform object query in an efficient compressed bitmap data table to obtain a first efficient compressed bitmap of an object set satisfying the corresponding query condition, the efficient compressed bitmap data table comprising efficient compressed bitmaps of a plurality of initial object sets; the initial object sets being determined by classifying objects according to attribute values; a combination module configured to combine the query codes corresponding to each of the at least two query conditions according to the Boolean logic relationship between the at least two query conditions to obtain a combination code; the combination code being used to perform logical operation on the first efficient compressed bitmap according to the Boolean logic relationship between the at least two query conditions to obtain a target efficient compressed bitmap; and the target efficient compressed bitmap being used to determine a query result.

[0006] In some embodiments of the present application, based on the foregoing scheme, the combination module comprises: an obtaining unit configured to obtain an efficient compressed bitmap function corresponding to the Boolean logic relationship according to the Boolean logic relationship between the at least two query conditions; and a combination unit configured to combine the query codes corresponding to each of the at least two query conditions according to the efficient compressed bitmap function corresponding to the Boolean logic relationship to obtain the combination code.

[0007] In some embodiments of the present application, based on the foregoing scheme, the query information further comprises classification statistical information, the classification statistical information indicating a target information item that needs to be counted according to attribute values; and the data processing apparatus further comprises: a classification statistical indication code generating module configured to generate a classification statistical indication code according to the classification statistical information; and an updating module configured to update the combination code according to the classification statistical indication code, the updated combination code being used to classify the object set indicated by the target efficient compressed bitmap according to the attribute values of the target information item.

[0008] In some embodiments of the present application, based on the foregoing scheme, the query request comprises a query task identifier, which is generated by an initiator of the query request upon detecting an input operation for query information; the data processing apparatus further comprises a query information acquisition module configured to acquire the query information associated with the query task identifier from a designated information table; the initiator of the query request stores the query task identifier and the detected query information in association in the designated information table upon detecting the input operation for query information; a combination code storage module configured to store a combination code corresponding to the query information in association with the query task identifier in the designated information table; and a storage instruction information return module configured to return storage instruction information to the initiator of the query request, so that the initiator of the query request acquires and executes the combination code from the designated information table according to the storage instruction information.

[0009] In some embodiments of the present application, based on the foregoing scheme, the object is a user; the data processing apparatus further comprises a data acquisition module configured to acquire user operation data and user attribute data, wherein the user operation data is used to indicate the interactive behavior of the user on the user interface of the product; an efficient bitmap compression module configured to compress the first user identifier corresponding to the user according to the attribute value of each field in the user operation data and the attribute value of each field in the user attribute data, to obtain an efficient compression bitmap of the corresponding attribute value; and a storage module configured to store the obtained efficient compression bitmap and the corresponding attribute value in the efficient compression bitmap data table.

[0010] In some embodiments of the present application, based on the foregoing scheme, the high-efficiency compressed bitmap data table comprises a second data table and a third data table; the high-efficiency bitmap compression module comprises: a generation unit configured to generate a user operation table according to the user operation data and generate a user attribute table according to the user attribute data, wherein the user operation table and the user attribute table each comprise a first user identifier corresponding to each user; a second high-efficiency compressed bitmap generation unit configured to perform high-efficiency bitmap compression on the first user identifiers corresponding to the same attribute value in the user operation table according to the attribute value of each field in the user operation table, to obtain a second high-efficiency compressed bitmap associated with the corresponding field and the corresponding attribute value; and a third high-efficiency compressed bitmap generation unit configured to perform high-efficiency bitmap compression on the first user identifiers corresponding to the same attribute value in the user attribute table according to the attribute value of each field in the user attribute table, to obtain a third high-efficiency compressed bitmap associated with the corresponding field and the corresponding attribute value; in this embodiment, a storage module comprises: a first storage unit configured to store the second high-efficiency compressed bitmap and the corresponding field and the corresponding attribute value in the second data table; and a second storage unit configured to store the third high-efficiency compressed bitmap and the corresponding field and the corresponding attribute value in the third data table.

[0011] In some embodiments of the present application, based on the foregoing scheme, the user operation data and the user attribute data comprise a user identifier; the data processing apparatus further comprises: a first user identifier generation unit configured to, if the user identifier is in a format that does not support high-efficiency bitmap compression, generate a first user identifier corresponding to each user identifier in a specified format that supports high-efficiency bitmap compression; in this embodiment, the generation unit comprises: a user operation table generation unit configured to generate the user operation table according to a first preset field, the user operation data, and the first user identifier corresponding to each user identifier; and a user attribute table generation unit configured to generate the user attribute table according to a second preset field, the user attribute data, and the first user identifier corresponding to each user identifier.

[0012] In some embodiments of the present application, based on the foregoing scheme, the data processing apparatus further comprises: an accumulation module configured to accumulate the first user identifiers to obtain an accumulated number; a first user identifier acquisition module configured to, if the accumulated number reaches a set number threshold, acquire the first user identifier corresponding to a user with the longest inactive duration; and an allocation module configured to allocate the first user identifier corresponding to the user with the longest inactive duration to a user to be generated with a next first user identifier.

[0013] In some embodiments of the present application, based on the foregoing scheme, the high-efficiency compressed bitmap data table further comprises a user group data table; the data processing apparatus comprises: a user identification set determination module, configured to determine a user identification set corresponding to a user group according to user identifications corresponding to users in the user group; a first user identification set determination module, configured to determine a first user identification corresponding to each user identification in the user identification set according to a mapping relationship between the user identification and the first user identification, to obtain a first user identification set corresponding to the user group; a compression module, configured to perform high-efficiency bitmap compression on the first user identifications in the first user identification set, to obtain a high-efficiency compressed bitmap corresponding to the user group; and a user group data table storage module, configured to store the high-efficiency compressed bitmap corresponding to the user group and a user group identification corresponding to the user group in the user group data table.

[0014] In some embodiments of the present application, based on the foregoing scheme, the query request comprises at least one of an event analysis request, a retention analysis request, a funnel analysis request, and a user path analysis request.

[0015] According to an aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; a memory, the memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the data processing method described above.

[0016] According to an aspect of an embodiment of the present application, a computer readable storage medium is provided, having computer readable instructions stored thereon, the computer readable instructions being executed by a processor to implement the data processing method described above.

[0017] In the scheme of the present application, the query request corresponding limited information is split into at least two query conditions, and a query code corresponding to each query condition is generated, the first high-efficiency compressed bitmap satisfying the corresponding query condition is obtained by respectively executing the query code corresponding to each query condition, and then the combination of the query codes is performed according to the Boolean logical relationship between the two query conditions, so that the obtained combination code can perform Boolean logical operation on the first high-efficiency compressed bitmap corresponding to at least two query conditions according to the Boolean logical relationship between the at least two query conditions, to obtain the target high-efficiency compressed bitmap of the object set satisfying the condition limited by the condition limited information. The present scheme realizes the splitting of the condition limited by the condition limited information, and performs block query according to the corresponding query code based on the splitting into at least two query conditions, which reduces the query difficulty and complexity compared with directly querying according to the condition limited by the condition limited information, and improves the query efficiency.

[0018] Moreover, in the scheme, the data is pre-compressed by the efficient bitmap and stored in the efficient compressed bitmap data table, which greatly reduces the data storage pressure, and in the data query process, the logical operation is based on the efficient compressed bitmap in the efficient compressed bitmap data table. Compared with the data table operation, the operation speed of the logical operation on the efficient compressed bitmap is faster and the efficiency is higher, so the overall data query efficiency can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application. It is apparent that the accompanying drawings are only some embodiments of the present application, and other drawings can be obtained according to the drawings without creative labor for those skilled in the art.

[0020] Figure 1 A and Figure 1 B shows a schematic diagram of an implementation environment to which the technical scheme of the embodiments of the present application can be applied.

[0021] Figure 2 is a flowchart of a data processing method according to an embodiment of the present application.

[0022] Figure 3 is a flowchart of a data processing method according to another embodiment of the present application.

[0023] Figure 4 is a flowchart of generating an efficient compressed bitmap data table according to an embodiment of the present application.

[0024] Figure 5 shows a system architecture diagram suitable for implementing the embodiments of the present application.

[0025] Figure 6 is a flowchart of a data processing method according to a specific embodiment of the present application.

[0026] Figure 7 is an interface diagram of a new event analysis according to a specific embodiment.

[0027] Figure 8 is an interface diagram of a new retention analysis according to an embodiment of the present application.

[0028] Figure 9 is an interface diagram of a new funnel analysis according to an embodiment of the present application.

[0029] Figure 10 A-10C is an interface diagram of a new user group according to an embodiment of the present application.

[0030] Figure 11 is a block diagram of a data processing apparatus according to an embodiment of the present application.

[0031] Figure 12 A structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown. DETAILED DESCRIPTION

[0032] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0033] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the

[0034] The block diagrams in the drawings show only the functional entities and not necessarily the physical separate entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flow diagrams shown in the drawings are merely examples and not necessarily to be construed as including all content and operations / steps, nor necessarily to be performed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so that the actual order of performance can be changed according to actual conditions.

[0036] It should be noted that "a plurality" means two or more in the present text. The association relationship between the associated objects described by "and / or" means that there can be three kinds of relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally means that the associated objects before and after it are in an "or" relationship.

[0037] Roaring bitmap: namely, efficient bitmap compression, is an efficient bit storage method, and its main idea is: 32-bit unsigned integer is divided into containers according to the high 16 bits (there can be at most 216=65536 containers), and when storing data, the container is found according to the high 16 bits of the data, and the low 16 bits are put into the container.

[0038] Figure 1 A and Figure 1 B shows a schematic diagram of an implementation environment to which the technical solutions of the embodiments of the present application can be applied.

[0039] As Figure 1 A shows, the implementation environment includes a first device 110 and a second device 120, wherein the first device 110 is in communication connection with the second device 120, which can be wireless or wired network connection. The first device 110 can be a terminal device such as a tablet computer, a notebook computer and a desktop computer, and the second device 120 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud computing, cloud storage, middleware services, big data and other basic cloud computing services, etc., which is not limited here.

[0040] The first device 110 can run a client program for data analysis, and the second device 120 provides services for the client program. The first device 110 can initiate a query request to the second device based on the user interface provided by the client program, and the second device 120 can perform data query according to the method of the present application after obtaining the query information indicated by the query request, obtain a target efficient compression bitmap of the object satisfying the condition limited in the query information, and then determine the query result for the query request according to the target efficient compression bitmap. In some embodiments of the present application, the second device can return the target efficient compression bitmap as the query result to the first device.

[0041] Since the second device 120 performs data query based on the efficient compression bitmap in the efficient compression bitmap set, the second device 120 or a database accessible by the second device 120 stores the efficient compression bitmap data table. The efficient compression bitmap data table includes efficient compression bitmaps of a plurality of initial object sets. The initial object set is determined by classifying objects according to attribute values. Generally, the corresponding object is represented by an object identifier, so the initial object set can be understood as a set of object identifiers, and then the efficient compression bitmap can be understood as efficient bitmap compression of the object identifiers in the initial object set.

[0042] The object can be a user of a software product, such as a WeChat user, a user following a public account, or a device, such as a device (e.g., an electric meter, a transformer, etc.) in a power grid system, a node device in a blockchain system, a server in a server cluster, etc.

[0043] Figure 1 B is a schematic diagram of another implementation environment according to an embodiment of the present application, which is compared with Figure 1 A, Figure 1 The third device 130 is used to provide data for data processing, that is, object data related to the object. If the object is a user of a software application, the third device can be a terminal device of a client where the user is located, or an application server of the software application. If the object is a server in a server cluster, the third device 130 can be a server in the server cluster.

[0044] The third device 130 can report the object data of the object to the second device 120. The second device 120 classifies the objects according to attribute values of the objects based on the object data of the objects, determines initial object sets corresponding to the attribute values, and performs efficient bitmap compression on object identifiers in the initial object sets to obtain corresponding efficient compressed bitmaps. On this basis, the second device 120 can query the efficient compressed bitmaps corresponding to the initial object sets, or perform object analysis based on the efficient compressed bitmaps, or analyze the situation of the object.

[0045] In some embodiments of the present application, the second device 120 can also be a server in a cloud platform. Multiple second devices form a cloud platform, which can be used as a big data analysis platform to provide cloud services, cloud computing, cloud storage, middleware services, big data, and other basic cloud computing services. In this scenario, the second device 120 can obtain object data from multiple third devices 130, and then perform statistical analysis based on the object data. For example, if the collected object data is behavior data of a user in a software product, the statistical analysis can be user retention analysis, event analysis of the user in the software, etc., which is not limited here.

[0046] In some embodiments of the present application, the scheme of the present application can be applied to a blockchain system. In the blockchain system, the third device 130 can be a node, and the second device 120 can be an analysis node. The running logs and other running data of the third devices 130 are uploaded to the analysis node, which analyzes the running situation of other nodes to facilitate timely discovery of faulty or abnormal nodes in the blockchain system.

[0047] It should be noted that the data processing method provided in the present application is generally executed by the second device 120. However, in other embodiments of the present application, a terminal device with data processing capability can also have similar functions as the second device 120, and thus execute the data processing method provided in the present application.

[0048] The implementation details of the technical solutions of the embodiments of the present application are described in detail as follows:

[0049] Figure 2 A flowchart of a data processing method according to one embodiment of the present application is shown, which can be executed by a computer device with processing capability, such as a server, a terminal device, or a system composed of a terminal device and a server, which is not specifically limited here. Referring to Figure 2 The method includes at least steps 210 to 240, which are described in detail as follows:

[0050] In step 210, a query request is acquired, which indicates query information including condition limiting information.

[0051] The condition limiting information is used to limit the conditions of the objects to be queried. In specific embodiments, the condition limiting information can limit the attribute values corresponding to the objects. For example, if the objects to be queried are users using software product A and the age of the users is 18-28 years old, the conditions limited by the condition limiting information include "using software product A" and "age of 18-28 years old". Further, if software product A includes multiple versions, the version of software product A, version number, etc. can be further specified.

[0052] An object can be related to multiple information items. If an information item is referred to as an information item, each object has a corresponding value under an information item, which is referred to as the attribute value of the object under the information item. For example, in the above example, age can be referred to as an information item, and "18-28 years old" is the attribute value range of the user under the age information item. The software product used can be referred to as an information item, and "software product A" is the attribute value of the user under the software product information item.

[0053] The conditions limited by the condition limiting information can be specified by information items, and the attribute values or attribute value ranges of the specified information items are limited. The information items specified in the condition limiting information can include multiple information items.

[0054] The condition limiting information can be a search formula input by a user, or the content input by the user in each query option in a user interface, wherein the query option is used for information item specification, and the input content is used to limit the attribute values or attribute value ranges of the corresponding information.

[0055] At step 220, the condition limiting information is parsed to determine at least two query conditions corresponding to the condition limiting information and a Boolean logical relationship between the at least two query conditions.

[0056] At step 220, the condition defined by the condition limiting information is split into at least two query conditions by parsing the condition limiting information, thereby simplifying the data query process.

[0057] In the case where the condition limiting information is specified by the application scenarios of the attribute values or attribute value ranges of the information items, at step 220, the at least two query conditions are obtained by reorganizing the multiple information items defined in the condition limiting information, and the Boolean logical relationship between the at least two query conditions is determined correspondingly. The Boolean logical relationship can include logical and, logical or, logical not, logical or not, logical and not, etc.

[0058] Each query condition includes the attribute value or attribute value range of at least one specified item in the condition limiting information. In some embodiments of the present application, there can also be a common limiting condition in the at least two query conditions, that is, each query condition includes a limitation that is different from other query conditions, and also includes the common limiting condition.

[0059] At step 230, a query code corresponding to each of the at least two query conditions is generated respectively; the query code is used to query an object in the high-efficiency compressed bitmap data table to obtain a first high-efficiency compressed bitmap of an object set satisfying the corresponding query condition, and the high-efficiency compressed bitmap data table includes high-efficiency compressed bitmaps of a plurality of initial object sets.

[0060] The object data corresponding to an object can include multiple information items, wherein each information item can be regarded as an information item, and each information item can be represented by a field. The value of the field corresponding to the information item is referred to as an attribute value, that is, the attribute value corresponding to the information item.

[0061] The object data can include attribute values of multiple information items, that is, the information items associated with an object can include multiple information items. For example, for a product user, the information items related to the user can include user age, gender, registration time, product, control triggered in the product, time of triggering the control, etc.

[0062] According to the attribute value corresponding to each information item in the object data, objects corresponding to the same attribute value are grouped into the same initial object set, thereby obtaining a plurality of initial object sets. It can be understood that an object can be identified by an object identifier, and therefore, the initial object set can be understood as a set of object identifiers corresponding to a plurality of objects having the same attribute value. Furthermore, the efficient compression bitmap of the initial object set is obtained by efficiently compressing the object identifiers in the initial object set. The efficient compression bitmap can also be referred to as a bitmap element, and therefore, the efficient compression bitmap of the initial object set refers to a bitmap element generated according to the object identifiers in the initial object set.

[0063] It is worth mentioning that, because efficient bitmap compression has a data format limitation, only unsigned integer data (for example, Uint16 (unsigned 16-bit integer), Uint32 (unsigned 32-bit integer)) can be efficiently compressed, and therefore, before efficiently compressing the object identifiers in the initial object set, if the object identifier is not in a format that supports efficient bitmap compression, the object identifier needs to be converted in format so that the object identifier after format conversion supports efficient bitmap compression, or a new object identifier in a format that supports efficient bitmap compression is generated to efficiently compress the new object identifier corresponding to the initial object set.

[0064] The attribute value referred to in the process of dividing the initial object set can be the attribute value of one information item or the attribute values of a plurality of information items, and can be set according to actual needs.

[0065] In some embodiments of the present application, in order to facilitate data statistical analysis after data query, the objects can also be classified according to the attribute values of a plurality of set information items, which can be set according to actual needs or the information items corresponding to the query options in the query interface. For example, in the query interface, the query options include a query option for specifying an event, and the objects can be classified according to the attribute values of the information items corresponding to the query option to determine the initial object set. This not only provides a data basis for data query, but also ensures the utilization efficiency of the efficient compression bitmap corresponding to the initial object set, thereby avoiding storing too many efficient compression bitmaps in the efficient compression bitmap data table.

[0066] The efficient compression bitmap data table is used to store the efficient compression bitmaps of the initial object sets. Because the initial object sets are determined by classifying the objects according to attribute values, the same attribute values corresponding to the initial object sets and the efficient compression bitmaps of the initial object sets can be stored in association, thereby facilitating the acquisition of the efficient compression bitmap based on the corresponding attribute value in the query process.

[0067] The first efficient compressed bitmap refers to an efficient compressed bitmap of object identifiers corresponding to objects satisfying a query condition. Based on the efficient compressed bitmap of the initial object set, Boolean logical operations can be performed on the efficient compressed bitmap of each initial object set according to the attribute values of each information item defined in the query condition and the Boolean logical relationship between the information items, so as to obtain the efficient compressed bitmap of object identifiers corresponding to objects satisfying the query condition.

[0068] After the query code corresponding to each query condition is generated respectively, the first efficient compressed bitmap with the same number as the query conditions can be obtained by executing the query code corresponding to each query condition.

[0069] In some embodiments of the present application, since the efficient compressed bitmap of the initial object set is stored in the efficient compressed bitmap data table, the query code can be generated according to the syntax rules of the SQL (Structured Query Language) of the database system.

[0070] In step 240, the query code corresponding to each query condition in the at least two query conditions is combined according to the Boolean logical relationship between the at least two query conditions, to obtain a combined code; the combined code is used to perform logical operations on the first efficient compressed bitmap according to the Boolean logical relationship between the at least two query conditions, to obtain a target efficient compressed bitmap; and the target efficient compressed bitmap is used to determine a query result.

[0071] The combined code obtained can call the first efficient compressed bitmap obtained by executing the query code corresponding to each query condition, and can perform Boolean logical operations on the first efficient compressed bitmap corresponding to each query condition in the at least two query conditions according to the Boolean logical relationship between the at least two query conditions.

[0072] In some embodiments of the present application, step 240 can include: obtaining an efficient compressed bitmap function corresponding to the Boolean logical relationship according to the Boolean logical relationship between the at least two query conditions; and combining the query code corresponding to each query condition in the at least two query conditions according to the efficient compressed bitmap function corresponding to the Boolean logical relationship, to obtain the combined code.

[0073] It can be understood that, in order to combine the query codes, a function for performing logical operations on the efficient compressed bitmap is constructed in advance, and the function for performing logical operations on the efficient compressed bitmap is referred to as an efficient compressed bitmap function.

[0074] Efficient bitmap functions such as bitmapAnd, bitmapOr, groupBitmapAnd, groupBitmapOr, bitmapAndCardinality, groupBitmapOrState, groupBitmapAndState, groupBitmapXor, and the like. Among them, the bitmapAnd function is to perform an AND operation on two bitmap objects (efficient bitmap compression), returning a new bitmap object; from the perspective of operation, it is an AND operation on a bit, and from the perspective of SQL, it is that both conditions are met. The bitmapOr function is used to perform an OR operation on two bitmap objects, returning a new bitmap object. The groupBitmapAnd function is to take the intersection of multiple bitmap objects, returning the number of elements in the resulting bitmap object. The groupBitmapOr function is to take the union of multiple bitmap objects and remove duplicates, returning the number of elements in the resulting bitmap object. The bitmapAndCardinality function is to perform an AND operation on two bitmap objects, returning the cardinality of the result bitmap (i.e., the number of elements in the result bitmap). The groupBitmapOrState function is used to perform an OR operation on multiple bitmap objects, returning a result bitmap object. The groupBitmapAndState function is used to perform an AND operation on multiple (two or more) bitmap objects, returning a result bitmap object. The groupBitmapXor function is used to remove duplicate values of any two bitmap objects from multiple bitmap objects, and to combine other values (each two bitmap objects are operated once), to obtain the final number. Of course, the above is only an exemplary example of efficient bitmap compression functions, and cannot be considered as a limitation of the scope of use of the present application.

[0075] In the process of executing the combination code, the first efficient bitmap compression obtained by executing each query code is called, so that the corresponding logical operation is performed on the first efficient bitmap compression based on the corresponding efficient bitmap compression function in the combination code, to obtain the corresponding target efficient bitmap compression. It can be understood that the efficient bitmap compression is the efficient bitmap compression corresponding to all objects satisfying the conditions defined in the query information.

[0076] In the scheme of the present application, the query request corresponding limited information is split into at least two query conditions, and the query code corresponding to each query condition is generated, the first efficient compressed bitmap satisfying the corresponding query condition is obtained by executing the query code corresponding to each query condition respectively, and then the combination of the query codes is performed according to the Boolean logical relationship between the two query conditions, so that the combined code can perform Boolean logical operation on the first efficient compressed bitmap corresponding to at least two query conditions according to the Boolean logical relationship between the at least two query conditions, and obtain the target efficient compressed bitmap of the object set satisfying the condition limited by the condition limited information. The present scheme realizes the splitting of the condition limited by the condition limited information, and based on the splitting into at least two query conditions, the block query is performed according to the corresponding query code. Compared with the direct query according to the condition limited by the condition limited information, the block query reduces the query difficulty and complexity, and improves the query efficiency.

[0077] Moreover, in the present scheme, the data is pre-processed for efficient bitmap compression and stored in the efficient compressed bitmap data table, which greatly reduces the data storage pressure. Moreover, in the data query process, the logical operation is performed based on the efficient compressed bitmap in the efficient compressed bitmap data table. Compared with the data table operation, the operation speed of the logical operation on the efficient compressed bitmap is faster and the efficiency is higher. Therefore, the data query efficiency can be improved as a whole.

[0078] In some embodiments of the present application, the query information further includes classification statistical information, and the classification statistical information indicates the target information item that needs to be counted according to the attribute value. In the present embodiment, the data processing method further includes generating a classification statistical indication code according to the classification statistical information. After step 240, the data processing method further includes updating the combined code according to the classification statistical indication code, and the updated combined code is used to classify the object set indicated by the target efficient compressed bitmap according to the attribute value of the target information item.

[0079] In some application scenarios, after the object set satisfying the condition limited by the query information is queried, further classification statistics of the data according to the attribute value is needed.

[0080] For example, if it is intended to know the number of users logging in product B on date A, it is further intended to know the number of users logging in each region on date A. Therefore, the classification statistical information can be used to indicate the classification of the users logging in product B on date A according to the region to which the user belongs, so as to count the number of users logging in product B in each region on date A. In this example, the information item that needs to be expanded according to the attribute value is the region to which the user belongs.

[0081] The classification statistic indication code can also be SQL code. The query code is combined into the combined code to obtain the combined code after the combination.

[0082] The combined code is updated by the classification statistic indication code, so that by executing the updated combined code, on one hand, a target efficient compressed bitmap of a target object set satisfying the condition limitation information can be obtained, and on the other hand, efficient compressed bitmaps of sub-object sets corresponding to attribute values in the target object set can be obtained. Further, by means of a function in the efficient compressed bitmap function for counting the cardinality of a bitmap object, the number of objects satisfying the condition information and the number of objects in the sub-object sets corresponding to the attribute values in the target object set can be obtained.

[0083] In some embodiments of the application, after the number of objects satisfying the condition information and / or the number of objects in the sub-object sets corresponding to the attribute values in the target object set are obtained, visual display can be performed, so as to facilitate the user to intuitively understand the query result.

[0084] Figure 3 is a flowchart of a data processing method according to an embodiment of the application, and Figure 3 In the embodiment shown in

[0085] Referring to Figure 3 The client first performs step 310 to obtain query information. An interactive interface for inputting query information is provided in the client, in which the user can input conditions for limiting the objects to be queried. Further, in the interactive interface, the user can specify information items and attribute values of the information items, for example, in Figure 3 the user can specify events (information items) and attribute values of the events by means of the control "+ Add Event", and specify other information items and attribute values of the other information items by means of the control "+ Add Filter Condition"; the user can also specify the attribute value (i.e. a specific date) of the information item "Date" in the option "Select Date". By detecting the operation triggered by the user in the interactive interface, the query information input by the user can be determined.

[0086] In some embodiments, the query information can be information indicating the control triggered by the user in the interactive interface and the input in the option. The server can determine the information items and attribute values specified by the user according to the control triggered by the user in the query information and the input in the option, and further determine the conditions of the limited objects.

[0087] Step 320, generating a query task identifier. The client generates the task identifier after detecting a new input operation for querying. The query task identifier and the query information are stored in the designated information table in association. The client and the server share the designated information table, so that the client and the server can access the designated information table.

[0088] Step 330, sending a query request. The client sends the query request to the server based on the generated query task identifier, wherein the query request carries the query task identifier.

[0089] After the server receives the query request, step 340 is performed to read the query information. Specifically, the server obtains the query information associated with the query task identifier from the designated information table according to the query task identifier carried in the query request. For example, the query information is obtained from the designated information table by get_task_meta according to the query task identifier.

[0090] Step 350, generating a combination code. The server can generate the combination code according to the process of steps 210-240 of the above embodiment after obtaining the query information; further, if the query information further includes classification statistical information, the classification statistical indication code is further generated according to the classification statistical information, and the combination code is updated according to the classification statistical indication code.

[0091] After generating the combination code, the server stores the combination code corresponding to the query information in the designated information table in association with the query task identifier, and performs step 360 to return storage indication information. The storage indication information is used to feed back the state of successfully storing the combination code in the designated information table.

[0092] After the client receives the storage indication information, step 370 is performed to read the combination code, and the combination code associated with the query task identifier is read from the designated information table according to the query task identifier. In specific embodiments, a program for reading the combination code from the designated information table is deployed in the client, for example, get_task_sql, to correspond to the execution of step 370 by running the program.

[0093] Step 380, executing the combination code. By executing the combination code, the target efficient compression bitmap corresponding to the limit information can be obtained; further, if the classification statistical indication code is combined in the combination code, the object set indicated by the target efficient compression bitmap according to the attribute value of the target information item can be further classified, and the number of efficient compression bitmaps corresponding to each attribute value of the target information item and / or objects is counted.

[0094] Step 390, page rendering. The client can perform page rendering, and can perform visual display of the query result, for example, the number of elements in the target high-efficiency compressed bitmap, that is, the number of objects meeting the condition limiting information, and the number of objects corresponding to each attribute value of the target information item in the objects meeting the condition limiting information.

[0095] In this embodiment, the client executes the combined code generated by the server to obtain the query result meeting the query information, which reduces the service pressure of the server. Moreover, since the combined code executed by the client is based on logical operation on the high-efficiency compressed bitmap in the high-efficiency compressed bitmap data table, the operation amount is greatly reduced compared with the data table without high-efficiency compression, and therefore, the client with limited processing capability can also execute the combined code, thereby ensuring the speed and efficiency of data query.

[0096] The scheme of the present application can be applied to an analysis platform, in which the method of the present application can be used for retention analysis, event analysis, user group analysis, funnel analysis, etc. of users of a software application, that is, the query request in the present application can be initiated based on retention analysis, user group analysis, funnel analysis, or event analysis, in other words, the query request can be at least one of an event analysis request, a retention analysis request, a funnel analysis request, and a user group analysis request.

[0097] Retention analysis is an analysis model for analyzing user participation / activeness, and investigates how many people of the users performing initial behavior will perform subsequent behavior. Event analysis refers to recording the process of a business by studying the behavior events of users, such as user registration, browsing product detail pages, ordering, payment, etc., and mining the reasons, interaction, etc. behind the behavior events by studying all factors associated with the occurrence of the events. User group analysis refers to analyzing the behavior of users in a group, in which users with the same attributes are divided into a group based on attributes such as historical behavior path, behavior characteristics, preferences, etc. of the users, and subsequent analysis is performed. Funnel analysis is used to analyze the conversion and loss of the behavior path of users at each step, to perform multi-dimensional analysis on the paths with more loss, and to find out the leakage points to improve conversion. Funnel analysis is a process-based data analysis, which can scientifically reflect the behavior state and the conversion and loss at each stage from the starting point to the end point.

[0098] In this application scenario, the object in the above embodiment is a user of a product, for example, a user of a news application, a user of a video application, a user of a shopping application, etc. Correspondingly, the initial object set is a user set determined by classifying users according to attribute values, and the user set includes user identifiers of users belonging to the same attribute value; and the high-efficiency compressed bitmap of the initial object set is obtained by high-efficiency bitmap compression of the user identifiers in the initial object set.

[0099] In this application scenario, generally speaking, the data related to the user includes the user operation data of the dynamic behavior of the user in the product and the user attribute data representing the basic information of the user. The user can be classified in combination of the user operation data and the user attribute data, and then the efficient compression bitmap of each initial object set (which can be regarded as an initial user set) is determined, and the efficient compression bitmap in the efficient compression bitmap set is used as the data basis for analysis (such as retention analysis and event analysis).

[0100] In some embodiments of the present application, as shown in FIG. 4, before step 220, the efficient compression bitmap data table needs to be generated in advance, which can include the following steps 410-430. Figure 4

[0101] Step 410, obtaining user operation data and user attribute data, the user operation data being used to indicate the interactive behavior of the user on the user interface of the product.

[0102] The user operation data can be obtained by setting a buried point and issuing the corresponding buried point to the client where the user is located, so that when the user triggers the operation in the user interface of the client for the buried point, the control information (such as control identifier, position information of the control in the page, etc.) of the control triggered by the user is reported to the server, and the data reported by the client where the user is located for the buried point can be used as the user operation data.

[0103] In some embodiments of the present application, the user operation data can be collected for one software application, or can be collected for multiple software applications, in other words, the user operation data can indicate the interactive behavior on the user interface of one product, or can indicate the interactive behavior on the user interfaces of multiple products.

[0104] The user attribute data can include the registration information of the user in the software application, such as user age, birth date, location, registration time, user nickname, zodiac, etc.

[0105] Step 420, performing efficient bitmap compression on the first user identifier corresponding to the same attribute value according to the attribute value of each field in the user operation data and the attribute value of each field in the user attribute data, to obtain the efficient compression bitmap of the corresponding attribute value.

[0106] Step 430, storing the obtained efficient compression bitmap and the corresponding attribute value in the efficient compression bitmap data table.

[0107] ​In some embodiments of the present application, the data table can be constructed according to the attribute values of each field in the user operation data and the attribute values of each field in the user attribute data. In this embodiment, the high-efficiency compressed bitmap data table includes a second data table and a third data table. In this embodiment, please refer to Figure 4 As shown, step 420 further includes steps 421-424.

[0108] Step 421 generates a user operation table according to the user operation data.

[0109] Step 422 generates a user attribute table according to the user attribute data. The user operation table and the user attribute table include the first user identifier corresponding to each user.

[0110] To generate the user operation table and the user attribute table, the definition of each field in the user operation table and the user attribute table is performed first, so that the information items in the user operation data are associated with the fields in the user operation table, and the information items in the user attribute data are associated with the fields in the user attribute table. Then, the attribute values of each information item in the user operation data are assigned to the corresponding field of the information item to obtain the user operation table. The attribute values of each information item in the user attribute data are assigned to the corresponding field of the information item to obtain the user attribute table.

[0111] In some embodiments of the present application, the user attribute table and the user operation table are narrow tables, so that the number of fields defined in the user operation table and the user attribute table can be reduced. In the subsequent user classification process according to the attribute values of the fields, the number of categories can be reduced, and correspondingly, the number of high-efficiency compressed bitmaps in the initial user set is reduced, and the storage pressure is further reduced. In some embodiments of the present application, the fields in the user attribute table and the user operation table can be further defined as relatively unified fields.

[0112] In some embodiments of the present application, part of the information items in the user operation data and the user attribute data obtained in step 410 are not needed to be concerned in the subsequent data analysis process. Therefore, before step 420, the user operation data and the user attribute data can be cleaned and filtered to retain the attribute values of the information items needed to be involved in the subsequent analysis, and the attribute values of the information items not involved in the other product analysis are filtered out.

[0113] The user operation table and the user attribute table both include the first user identifier corresponding to the user, so that the information in the user operation table and the information in the user attribute table are associated through the first user identifier of the corresponding user in the two tables.

[0114] Step 423, according to the attribute value of each field in the user operation table, the first user identification corresponding to the same attribute value in the user operation table is compressed by high efficiency bitmap, and a second high efficiency compressed bitmap associated with the corresponding field and the corresponding attribute value is obtained. And step 424, according to the attribute value of each field in the user attribute table, the first user identification corresponding to the same attribute value in the user attribute table is compressed by high efficiency bitmap, and a third high efficiency compressed bitmap associated with the corresponding field and the corresponding attribute value is obtained.

[0115] In this embodiment, step 430 includes: step 431, storing the second high efficiency compressed bitmap and the corresponding field and the corresponding attribute value in the second data table; step 432, storing the third high efficiency compressed bitmap and the corresponding field and the corresponding attribute value in the third data table.

[0116] In some embodiments of the present application, the first user identification corresponding to the same attribute value can be compressed by high efficiency bitmap through the aggregation function groupBitmapState, that is, the aggregation function is used to aggregate multiple elements into a bitmap object (high efficiency compressed bitmap). Wherein the same attribute value can be an attribute value of an information item, or two or more attribute values, for example, the first user identification corresponding to the attribute value of 18-25 years old under the information item age in the user attribute table can be aggregated into a bitmap object, and a high efficiency compressed bitmap corresponding to the attribute value is obtained; for example, the user identification corresponding to the user whose age is 18-25 years old and whose registered place is North China can be aggregated into a bitmap object.

[0117] In some embodiments of the present application, in steps 423 and 424, in order to control the number of high efficiency compressed bitmaps of the high efficiency compressed bitmap data table, the users can be classified according to the attribute value of the specified field, and the first user identification corresponding to the same attribute value in the user operation table (and the user attribute table) under the specified field is compressed by high efficiency bitmap, without the need to compress the high efficiency bitmap for the attribute value under each field in the user operation table and / or the user attribute table. The specified field can be set according to actual needs, which is not limited here. For example, because in the analysis process, classification statistics based on user operation behavior and user operation time are involved, the field indicating user operation behavior and the field indicating user operation time can be used as the specified field.

[0118] It is worth mentioning that the object of high efficiency bitmap compression is unsigned integer data, so before high efficiency bitmap compression, it is necessary to ensure that the first user identification to be compressed by high efficiency bitmap is in the format that supports high efficiency bitmap compression, that is, unsigned integer data.

[0119] For the user using the software application, the user operation data is constantly updated as time goes on, while the user attribute data is generally updated at a lower frequency. In this embodiment, the user operation table is generated according to the user operation data, and the user attribute table is generated according to the user attribute data, so that the user operation table and the user attribute table can be updated respectively, without the need to update the user operation table and the user attribute table simultaneously when the user operation data or the user attribute data is updated.

[0120] In some embodiments of the present application, the user operation table can also be constructed only for the user operation data, so that the first user identifiers corresponding to the same attribute value in the user operation table are efficiently bitmap compressed based on the attribute values of the fields in the user operation table. In some other embodiments of the present application, the user operation data and the user attribute data can also be combined into the same data table, so that the first user identifiers corresponding to the same attribute value are efficiently bitmap compressed based on the attribute values of the fields in the data table, which can be set according to actual needs.

[0121] In some embodiments of the present application, the user operation data and the user attribute data include user identifiers. Before step 421 and step 422, the method further includes: if the user identifier is in a format that does not support efficient bitmap compression, generating the first user identifier corresponding to each user identifier in a specified format that supports efficient bitmap compression.

[0122] In this embodiment, step 421 further includes: generating the user operation table according to the first preset field, the user operation data and the first user identifier corresponding to each user identifier; and step 422 further includes: generating the user attribute table according to the second preset field, the user attribute data and the first user identifier corresponding to each user identifier.

[0123] The format of the generated first user identifier is a format that supports efficient bitmap compression. The first preset field refers to a field defined for the user operation table; and the second preset field refers to a field defined for the user attribute table. Through the above process, it can be ensured that the first user identifier corresponding to each user in the user attribute table and the user operation table is in a format that supports efficient bitmap compression.

[0124] In some embodiments of the present application, in order to avoid affecting other data after the user identifier is replaced by the first user identifier, other fields in the user operation table and the user attribute table that are different from the field corresponding to the user identifier can be defined to represent the first user identifier.

[0125] In some embodiments of the present application, the method further comprises: accumulating the first user identifiers to obtain an accumulated number; if the accumulated number reaches a set number threshold, obtaining the first user identifier corresponding to the user with the longest inactive duration; and assigning the first user identifier corresponding to the user with the longest inactive duration to the next user to be generated with a first user identifier.

[0126] In the present embodiment, the elimination mechanism is adopted to eliminate the users that have not been active for a long time, and the first user identifier of the eliminated user is reused, which can ensure that the total number of first user identifiers is maintained within a range not exceeding the set number threshold, thereby ensuring that the space size of the efficiently compressed bitmap in the bitmap data table is within a stable range, and avoiding the problem of reduced computing performance caused by the sparseness of the efficiently compressed bitmap due to too many first user identifiers. Moreover, since the eliminated user is the user with the longest inactive duration, and the behavior analysis is generally performed based on the users that have been active in the current time or a period of time close to the current time, reusing the first user identifier of the eliminated user will not affect the data analysis performed.

[0127] In some embodiments of the present application, the bitmap data table further comprises a user group data table; and the data processing method further comprises: determining a user identifier set corresponding to a user group according to the user identifiers corresponding to the users in the user group; determining the first user identifier corresponding to each user identifier in the user identifier set according to the mapping relationship between the user identifiers and the first user identifiers, to obtain a first user identifier set corresponding to the user group; performing efficient bitmap compression on the first user identifiers in the first user identifier set to obtain an efficiently compressed bitmap corresponding to the user group; and storing the efficiently compressed bitmap corresponding to the user group and the user group identifier corresponding to the user group in the user group data table.

[0128] In the present embodiment, based on the obtained efficiently compressed bitmap corresponding to the user group, the users in the user group can be used to perform retention analysis, event analysis, funnel analysis, etc. Compared with performing product analysis such as retention analysis, event analysis, and funnel analysis based on all users using the product, performing product analysis based on the users in the user group is equivalent to narrowing the user range.

[0129] Next, the method of the present application will be further described in combination with a specific embodiment.

[0130] Figure 5 is a system architecture diagram according to a specific embodiment of the present application. As shown in Figure 5 the system architecture comprises an offline storage module 510, a computing engine 520, and an application layer 530. Each terminal (for example, a mobile phone, a tablet computer, a computer, etc.) can be connected to the system architecture via the application layer 530. Figure 1The third device in B) reports data (user operation data, user attribute data) to the offline storage module 510, which stores the data offline. The offline storage module 510 can be constructed using Hadoop HDFS (Hadoop Distributed File System), and can further clean and normalize the data to obtain structured data.

[0131] The computing engine 520 is configured to process the data in the offline storage module 510, and includes a Spark computing engine and a Clickhouse query engine. Clickhouse is an open-source columnar database for online analytical processing (OLAP), which provides a rich bitmap function group. Therefore, the bitmap function in Clickhouse can be called to perform operations based on efficient compression in the efficient compression bitmap database. Under this system architecture, the efficient compression bitmap of users meeting the specified conditions can be stored in Clickhouse, and the Clickhouse query engine can be used to query the relevant information of each user in the efficient compression bitmap. The Spark computing engine can be used for portrait analysis and user preference analysis of a user group.

[0132] In the application layer 530, data overview functions and analysis tools are provided. For the data overview function, the number of users under a preset statistical dimension can be counted, such as the dimensions of new, online, active, and retention. The new dimension refers to the number of new users in each time period, the online dimension refers to the number of online users in each time period, the active dimension refers to the number of users accessing the application in each time period, and the retention dimension refers to the number of retained users in each time period. The analysis tools can be used for event analysis, funnel analysis, retention analysis, and user group analysis.

[0133] Figure 6 is a flowchart of a data processing method according to an embodiment of the present application. The data processing method can be implemented by the system architecture shown in Figure 5 , as shown in Figure 6 , includes the following steps 610-674. In step 610, the terminal reports data to HDFS.

[0134] The reported data includes user operation related data. The main fields include time, user id, channel, version, operation event id, and platform. The reported data can be as shown in Table 1. In Table 1, "Timestamp" represents time, and "uid" represents user identification.

[0135] Table 1

[0136]

[0137] Step 620, data preprocessing.

[0138] The data stored in the HDFS is cleaned and filtered by the hive tool of the HDFS system (a data warehouse tool based on Hadoop), and the attribute values of important fields required in subsequent data processing are extracted therefrom, such as the first channel of user registration, whether the user is a new user of the day, the latest version used by the user, and the like.

[0139] Step 630, generating product internal attribute table and product external attribute table.

[0140] In the embodiment, the user-related data is divided into two parts, one part is used to reflect the user's behavior in the product, and this part of data is referred to as user operation data, such as the version of the product used by the user, the channel of downloading the software product, and the like; and the other part is used to reflect the basic portrait information of the user, and this part of data is referred to as user attribute data, such as gender, age, and the like. Then, the product internal attribute table is generated according to the user operation data, and the product external attribute table is generated according to the user attribute data.

[0141] In the embodiment, the fields in the product external attribute table and the product internal attribute table are defined according to a unified field, such as: category, attribute, value, uid, ds, product, and the like, wherein the category represents the category of the attribute, such as channel information, product behavior, and the like, the attribute represents the attribute, such as the first channel, the function point, and the like, the value represents the specific attribute value, such as the channel name, the operation event id, and the like; the uid represents the user identifier; the ds represents the time; and the product represents the product name.

[0142] The product internal attribute table includes the user identifier uid and the user product attribute (such as the operation event, the channel), and the like, which can be as shown in Table 2.

[0143] Table 2

[0144]

[0145] The product external attribute table includes the user identifier uid and the user basic portrait information, which can be as shown in Table 3.

[0146] Table 3

[0147]

[0148] Step 640, encoding the user identifier.

[0149] The user identification (uid) extracted from the data reported by the terminal is not in a format that can support efficient bitmap compression. Therefore, in order to enable subsequent efficient bitmap compression, the user is identified and encoded, specifically, the user identification (uid) of each user is mapped to an integer id that can support efficient bitmap compression. For ease of distinction, the integer id mapped by the user identification (uid) is referred to as the first user identification. The value type of the first user identification that can be selected includes: UInt8: [0, 255], UInt16: [0, 65535], and UInt32: [0, 4294967295].

[0150] According to the mapping relationship between the user identification and the first user identification, the identification encoding table shown in Table 4 can be obtained. In specific embodiments, the first user identification encoding can be performed sequentially, facilitating the statistics and allocation of the first user identification.

[0151] Table 4

[0152]

[0153] Step 650, generating a user operation table and a user attribute table.

[0154] Based on the mapping relationship between the user identification and the first user identification in the identification encoding table, the user identification in the product internal attribute table generated in step 630 is converted into the first user identification, obtaining the user operation table, as shown in Table 5 below.

[0155] Table 5

[0156]

[0157] The user identification in the product external attribute table generated in step 630 is converted into the first user identification, obtaining the user attribute table, as shown in Table 6 below.

[0158] Table 6

[0159]

[0160] Step 660, generating a second data table and a third data table.

[0161] The user operation table and the user attribute table generated in step 650 are imported from the hive tool into the clickhouse system, and then the user ids (first user identifiers) belonging to the same attribute value are efficiently bitmap compressed using the aggregation function groupBitmapState in the clickhouse system according to the attribute values of the fields in the user operation table and the user attribute table. The first user identifiers corresponding to the same attribute value in the user operation table are efficiently bitmap compressed to obtain a second data table, which can be shown in Table 7.

[0162] Table 7

[0163]

[0164] The first user identifiers corresponding to the same attribute value in the user attribute table are efficiently bitmap compressed to obtain a third data table, which can be shown in Table 8.

[0165] Table 8

[0166]

[0167] The field bmp identifies the efficient compression bitmap, and {1, 100, 2435} in the field bpm in Table 7 indicates that the first user identifiers 1, 100, and 2435 are stored in the form of bits in the data type AggregateFunction (groupBitmap, UInt32) in the clickhouse system, and the others are similar. AggregateFunction is a special data type provided by the clickhouse system, which can store intermediate state results in binary form. For a column field of the AggregateFunction type, data writing and querying are different from ordinary data. When writing data, the State function needs to be called, and when querying data, the corresponding Merge function needs to be called.

[0168] Based on the obtained second data table and third data table, data overview, product analysis, and user group analysis functions can be performed. Through the spark computing engine and the clickhouse query engine, an index system is constructed to help us understand the product status and discover index changes. After obtaining this information, we can analyze the product data and further analyze the reasons for the abnormality, such as whether the new abnormality is related to a certain channel through event analysis, and discovering problems in product functions through funnel analysis. Or do more exploratory analysis, such as user group analysis to discover the characteristics of different user groups, to facilitate the operation personnel to make operation decisions and intelligent operation.

[0169] The data overview function is used to provide the basic data system of the product, such as the overall user activity, new users, retained users, etc. In this part, user statistics can be performed according to the statistics indexes such as new, networking, active, retention, etc.

[0170] Specifically, based on the second data table and the third data table, the commonly used new, networking, active, retention indexes and common dimensions can be calculated using the bitmap function (efficient compression bitmap function) group provided by the Clickhouse system, and the calculation results can be saved in the table of the Clickhouse system. For example, the bitmapCardinality function can be used to calculate some commonly used active indexes, such as daily active, weekly active, monthly active, and these indexes according to different attributes. In addition, the bitmapAndCardinality, bitmapOrCardinality and other functions can be used to calculate the retention related indexes, and finally the visual display is performed to provide the analysis data of the commonly used indexes for the products.

[0171] Based on the generated second data table and the third data table, a product analysis tool can be further built, which includes an event tool, a funnel tool and a retention tool. The event tool is used for event analysis, the funnel tool is used for funnel analysis, and the retention tool is used for retention analysis. The event tool can specify events (operation behaviors), specify different filtering conditions, and view the number of users according to a certain dimension.

[0172] The client provides an interactive interface for event analysis, in which event specification, filtering condition specification, and dimension expansion (attribute value expansion) specification can be performed. Figure 6 Steps 671-674 show a flowchart for block query in event analysis, as shown in Figure 6 Based on the specified event, the specified condition (filtering condition) and the specified data expansion dimension, the client can upload the query information to the server to determine the query conditions: the specified event, the specified condition, and the classification statistics indication information is determined by the step of analyzing, and then step 671 is executed to calculate the user bitmap of the specified event; step 672 is executed to calculate the user bitmap of the specified condition, and step 673 is executed to calculate the user bitmap of each dimension value under the specified dimension. Then, based on the Boolean logical relationship between the query conditions, step 674 is executed to perform logical operation on the user bitmap using the bitmap function. Through step 674, the user bitmap (target efficient compression bitmap) satisfying the condition limitation information limited condition and the user bitmap corresponding to each attribute value after expansion according to the data expansion dimension can be obtained, and finally the result obtained by step 674 can be visualized.

[0173] Figure 7 is a schematic diagram of the interface of the newly built event analysis according to an embodiment, which includes an event analysis information specification area 710, an event analysis display setting area 720, and an event analysis result display area 730. The information specification area 710 provides a control for adding an event, that is, a “+ Add Event” control, where the event is reflected through the user’s behavior, Figure 7 “Networking” is selected in the event analysis information specification area 710; the area 710 also provides a control for adding a filtering condition, that is, a “+ Add Filtering Condition” control, and a control for date specification, Figure 7 The specified date is 20210216-20210221. The event analysis information specification area 710 also provides a control for adding a viewing dimension, that is, a “+ Add Dimension” control. The selected event, the added filtering condition, and the selected date added by the user form the condition limitation information indicated by the event analysis request. The viewing dimension added by the user can be regarded as the classification statistical information indicated by the event analysis request. Thus, after the client initiates the event analysis request, the server can correspondingly analyze the condition limitation information and decompose the condition limited by the condition limitation information into at least two query conditions.

[0174] Figure 7 The event analysis display setting area 720 in the interface in the event analysis result display area 730 is used to display the queried result according to the display mode selected by the user, for example, a trend chart, a column chart, a pie chart, a table, etc. can be displayed, and more dimensions can be set through the “Display Setting” option. Figure 7 In the event analysis result display area 730, the viewing dimension of “version” is selected, and the result display area exemplarily shows the user quantity trend chart of the event of “networking” in the time period of 20210216-20210221 for each day under the versions of 8.10.1 and 8.10.0.

[0175] In the scenario where the query request is an event analysis request, the server can decompose the condition limited by the condition limitation information into a specified event and a filtering condition in the analysis of the condition limitation information, so that the query process can be divided into: finding the user bitmap of the event according to the event selected by the user, finding the bitmap of the user meeting the condition according to the filtering condition, and finding the user bitmap corresponding to each dimension value under the dimension according to the filtered dimension, then performing logical operation through the rich bitmap function group provided by clickhouse to obtain the statistical result, and finally visualizing the trend chart by the front end.

[0176] Retention tools are analytical models used to analyze user engagement / activity levels, examining how many users who perform an initial action will engage in subsequent actions. In implementation, they differ slightly from event-based tools in the concepts of initial and retention behaviors, but both are essentially events, thus both aim to find a user bitmap for a specific event.

[0177] Figure 8 This is a schematic diagram of a new retention analysis interface according to an embodiment of this application. The interface includes a retention analysis information specification area 810 and a retention analysis result display area 820. For example... Figure 8 As shown, the retention analysis information specification area 810 provides options for specifying the starting behavior, specifying the retention behavior, adding filter conditions, specifying the date, and specifying the viewing dimensions. Similar to event analysis, the starting behavior, specified retention behavior, added filter conditions, and specified date specified by the user in the retention analysis information specification area 810 are equivalent to indicating the conditions limited by the condition constraint information indicated by the retention analysis request. The viewing dimensions added by the user can be regarded as the categorical statistics indicated by the retention analysis request.

[0178] The retention analysis results display area 820 is used to display the results of the retention analysis. This retention analysis results display area 820 provides the retention rate / retention change rate for each day in a specified time period, displayed in tables and trend charts.

[0179] The retention analysis process will now be described in detail with reference to a specific embodiment. If the user specifies the following information items:

[0180] product_id = “manager”; start_event = “E_Rocket_Slip”; remain_event = “E_Accelerate_Show”; condition = “Version Information_Version_Includes_8.12.0”; ds_condition = 20210416_20210422; date_format = “day”; attribute_name = “Population Attribute_Age”.

[0181] Wherein, product_id represents the product identifier, start_event represents the initial behavior, remain_event represents the retention behavior; condition represents the filter condition; ds_condition represents the time period, and date_format represents statistics by day.

[0182] After the service end receives the query request, the query information associated with the query task identifier is obtained, and the conditions limited in the query information (i.e. the conditions limited by product_id, start_event, remain_event, condition and ds_condition in the above) are parsed into two query conditions: 1) the starting behavior is operated and the filtering condition is met (i.e. the conditions limited by start_event, product_id, condition, ds_condition); 2) the product operates the retention behavior (i.e. the conditions limited by product_id, remain_event, and the specified retention time); the classification statistical indication information in the query information is also included, that is, according to the age.

[0183] On this basis, the method according to the present application is used for block query:

[0184] ①Query the specified user object, that is, query the user who has operated the starting behavior and meets the filtering condition, and set the efficient compression bitmap of the user in this part to object_user_bmp.

[0185]

[0186] In the above code, as is used to indicate an alias, for example, select bmp as condition_bmp indicates that the alias of bmp is condition_bmp. Union all is used to perform a set operation on two select statements, which includes repeated rows.

[0187] By executing the above SQL code, the efficient compression bitmap of the user who meets the conditions of product_id = “manager”; start_event = “E_Rocket_Slip” condition = “version information_version_contains_8.12.0” in each day in the time period of 20210416_20210422 can be queried.

[0188] ②Query the efficient compression bitmap of the user corresponding to each attribute value that needs to be expanded, and define it as:

[0189]

[0190] ③Query the efficient compression bitmap of the user of the subsequent multi-day retention behavior, and define it as:

[0191] remain_user_bmp.

[0192]

[0193] By executing this piece of code, we can determine the bitmap of users who used the product "product_id ='manager'" in the time period 20210417-20210429, 20210430-20210516, and the behavior "E_Accelerate_Show" in this time period.

[0194] ④Code combination

[0195]

[0196] )t

[0197] The datediff() function is used to calculate the date part of the time difference between the second date and the first date of the specified two dates.

[0198] The parseDateTimeBestEffort function is used to convert the time and date of the String type to the DateTime data type.

[0199] The Cast() function is used to explicitly convert an expression of a certain data type to another data type. The parameter of the Cast() function is an expression, which includes the source value and the target data type separated by the AS keyword. Its usage is: cast(expression AS data_type), where expression represents any valid SQServer expression, AS is used to separate the two parameters, and the data before AS is the data to be processed, and the data type after AS is the data type to be converted.

[0200] data_type refers to the data type provided by the target system, including bigint and sql_variant, and cannot use user-defined data types.

[0201] Inner join is used to display all matching records in two or more tables associated according to the association condition. Cross join is used to connect all rows of table A with all rows of table B respectively, and the number of returned records is the product of the number of records of the two tables. Using() is used for join query of two tables, which requires that the column specified by using() exists in both tables and is used as the condition for join.

[0202] In the above-mentioned combined code, the result attribute_user_bmp obtained in ①, the result attribute_user_bmp obtained in ②, and the result remain_user_bmp obtained in ③ are called to perform corresponding logical operations to obtain the final user bitmap, and the number of users is counted according to age.

[0203] The query of the event tool and the funnel tool is similar to the process, and the way of the chunked query well guarantees the accuracy and efficiency of the query under complex conditions, and improves the query efficiency.

[0204] The funnel tool is mainly used for analyzing the conversion and loss of each step in a multi-step process, and can find the key loss point in the step, can help us find the problems in the product, and provides the data related to the path conversion in the product.

[0205] Figure 9 Fig. 1 is a schematic diagram of an interface for creating a funnel analysis according to an embodiment of the present application. As shown in Fig. 1, the interface includes a funnel analysis information specification area 910 and a funnel analysis result display area 920. The funnel analysis information specification area 910 provides a control " + Add funnel step" for adding a funnel step, Figure 9 The funnel steps added in the funnel analysis information specification area 910 include "Networking" and "Active" steps, and further provide a control for adding a filtering condition and a control for date specification. The funnel analysis result display area 920 provides a table format display of the number of users counted in each funnel step, and a histogram style display of the number of users in each funnel step. Figure 9

[0206] Based on the product internal attribute table and the product external attribute table generated in step 630, user group analysis is performed. Specifically, the user group can be generated according to the product internal attribute table and the product external attribute table.

[0207] Figure 10 A- Figure 10 Fig. 2 is a schematic diagram of an interface for creating a user group according to an embodiment of the present application. In this embodiment, three ways of creating a user group are provided: custom user group, uploaded user group, and SQL user group. The corresponding interactive interface of the custom user group is shown in Fig. 2A. The user can specify a date and add a filtering condition to filter the users that meet the specified date and the added filtering condition as the users in the user group. Figure 10 Figure 10 ​​In A, an option of setting the user group name is also provided, so as to allow the user to customize the user group name. As time goes on, the data of the user is updated correspondingly, and correspondingly, the user satisfying the specified date and the added filtering condition can also change, i.e., the user in the user group is updated correspondingly. In the scheme of the embodiment, as shown in Figure 10 A, an option of selecting the user group updating mode is provided, for example, manual updating, i.e., the user group is updated correspondingly when the user triggers the updating operation. In other embodiments, it can also be automatic updating, for example, the condition of automatic updating is set in advance, and when the condition of automatic updating is met, the updating is automatically performed.

[0208] Figure 10 B shows an interface schematic diagram of creating a user group by uploading a user group, as shown in Figure 10 B, the interface provides an uploading window 1010 of the user group package, when the user drags the file indicating the user in the user group to the uploading window 1010 or clicks the "upload" control in the uploading window, the user indicated in the file can be added to the user group. The file uploaded through the uploading window can be a user identifier (first user identifier) for indicating the user to be added to the user group. Similarly, Figure 10 A, Figure 10 The interface shown in B also provides an option of customizing the user group name.

[0209] Figure 10 C shows an interface schematic diagram of creating a user group by using SQL, as shown in Figure 10 C, the interface provides a code input area 1020 of inputting the user group SQL, by inputting the SQL code for creating the user group in the code input area 1020, for example, the SQL code indicating which user identifier is added to the user group, the user group is created correspondingly by executing the SQL code. Further, as shown in Figure 10 C, in order to avoid the failure of creating by executing the input SQL code due to the error of the SQL code, the interface also provides an option of checking the SQL code, if the user triggers the "SQL check" control, the terminal can automatically check the SQL code input in the code input area 1020, and display the checking result in the content display area corresponding to the "SQL check result". Similarly, Figure 10 A, Figure 11 The interface of C also provides the options of customizing the user group name and selecting the user group updating mode.

[0210] Through the three manners, a user of the analysis platform can flexibly create a personalized user group, and further performs user analysis according to data of users in the user group, for example, analyzes a portrait and a preference of the user. In specific embodiments, the portrait and the preference of the user group can be calculated by using a spark computing engine, and the result is stored in a Clickhouse system. Finally, a client (for example, a WEB client) displays the result in a visualized manner according to the calculation result.

[0211] In the embodiment, the user group analysis tool and other product analysis tools can be further connected, that is, event analysis, retention analysis, funnel analysis and the like in a product are performed in the user group. On the basis of the product internal attribute table and the product external attribute table and the condition of the user in the user group defined by the user in the interface of the newly created user group, the user identifier of the user in the user group can be determined. Then, according to the mapping relationship between the user identifier indicated by the identifier coding table obtained in step 640 and the first user identifier, the first user identifier set corresponding to the user group can be obtained, and the bitmap compression is performed on the first user identifier set corresponding to the user group, to obtain the bitmap of the user group. The bitmap of the user group can be used as the data basis for product analysis, so that the product analysis is performed based on the attribute value of the user in the user group.

[0212] The device embodiment of the present application is introduced below, which can be used to execute the method in the above-mentioned embodiments of the present application. For details not disclosed in the device embodiment of the present application, please refer to the above-mentioned method embodiments of the present application.

[0213] Figure 11 is a block diagram of a data processing device according to an embodiment, as shown in Figure 12As shown, the data processing apparatus comprises: a query request acquisition module 1110, configured to acquire a query request, the query request indicating query information, the query information comprising condition limiting information; an analysis module 1120, configured to analyze the condition limiting information, and determine at least two query conditions corresponding to the condition limiting information and a Boolean logic relationship between the at least two query conditions; a query code generation module 1130, configured to generate a query code corresponding to each of the at least two query conditions respectively; the query code being used for object query in a high-efficiency compressed bitmap data table to obtain a first high-efficiency compressed bitmap of an object set satisfying the corresponding query condition, the high-efficiency compressed bitmap data table comprising high-efficiency compressed bitmaps of a plurality of initial object sets; the initial object set being determined according to classification of objects according to attribute values; and a combination module 1140, configured to combine the query codes corresponding to the respective query conditions in the at least two query conditions according to the Boolean logic relationship between the at least two query conditions, to obtain a combination code; the combination code being used for logical operation on the first high-efficiency compressed bitmap according to the Boolean logic relationship between the at least two query conditions, to obtain a target high-efficiency compressed bitmap; and the target high-efficiency compressed bitmap being used for determining a query result.

[0214] In some embodiments of the present application, the combination module 1140 comprises: an acquisition unit, configured to acquire a high-efficiency compressed bitmap function corresponding to the Boolean logic relationship according to the Boolean logic relationship between the at least two query conditions; and a combination unit, configured to combine the query codes corresponding to the respective query conditions in the at least two query conditions according to the high-efficiency compressed bitmap function corresponding to the Boolean logic relationship, to obtain the combination code.

[0215] In some embodiments of the present application, the query information further comprises classification statistical information, the classification statistical information indicating a target information item that needs to be counted according to attribute values; and the data processing apparatus further comprises: a classification statistical indication code generation module, configured to generate a classification statistical indication code according to the classification statistical information; and an updating module, configured to update the combination code according to the classification statistical indication code, the updated combination code being used for classifying the object set indicated by the target high-efficiency compressed bitmap according to the attribute values of the target information item.

[0216] In some embodiments of the present application, the query request comprises a query task identifier, the query task identifier being generated by an initiator of the query request upon detecting an input operation for the query information; the data processing apparatus further comprises a query information obtaining module configured to obtain the query information associated with the query task identifier from a designated information table; the initiator of the query request stores the query task identifier and the detected query information in association in the designated information table upon detecting the input operation for the query information; a combination code storing module configured to store a combination code corresponding to the query information in association with the query task identifier in the designated information table; and a storage indication information returning module configured to return storage indication information to the initiator of the query request, so that the initiator of the query request obtains and executes the combination code from the designated information table according to the storage indication information.

[0217] In some embodiments of the present application, the object is a user; the data processing apparatus further comprises a data obtaining module configured to obtain user operation data and user attribute data, the user operation data being used to indicate an interactive behavior of the user on a user interface of a product; a high-efficiency bitmap compression module configured to compress, according to attribute values of each field in the user operation data and attribute values of each field in the user attribute data, first user identifiers corresponding to the same attribute values to obtain a high-efficiency compressed bitmap of the corresponding attribute values; and a storage module configured to store the obtained high-efficiency compressed bitmap and the corresponding attribute values in a high-efficiency compressed bitmap data table.

[0218] In some embodiments of the present application, the high-efficiency compressed bitmap data table comprises a second data table and a third data table; the high-efficiency bitmap compression module comprises a generation unit configured to generate a user operation table according to the user operation data and a user attribute table according to the user attribute data, the user operation table and the user attribute table comprising first user identifiers corresponding to each user; a second high-efficiency compressed bitmap generation unit configured to compress, according to attribute values of each field in the user operation table, the first user identifiers corresponding to the same attribute values in the user operation table to obtain a second high-efficiency compressed bitmap associated with the corresponding field and the corresponding attribute value; and a third high-efficiency compressed bitmap generation unit configured to compress, according to attribute values of each field in the user attribute table, the first user identifiers corresponding to the same attribute values in the user attribute table to obtain a third high-efficiency compressed bitmap associated with the corresponding field and the corresponding attribute value; in this embodiment, the storage module comprises a first storage unit configured to store the second high-efficiency compressed bitmap and the corresponding field and the corresponding attribute value in the second data table; and a second storage unit configured to store the third high-efficiency compressed bitmap and the corresponding field and the corresponding attribute value in the third data table.

[0219] In some embodiments of the present application, the user operation data and the user attribute data comprise user identifiers; the data processing apparatus further comprises a first user identifier generation unit configured to generate, if the user identifiers are in a format that does not support efficient bitmap compression, first user identifiers corresponding to the user identifiers in a specified format that supports efficient bitmap compression; in this embodiment, the generation unit comprises a user operation table generation unit configured to generate a user operation table according to the first preset field, the user operation data and the first user identifiers corresponding to the user identifiers; and a user attribute table generation unit configured to generate a user attribute table according to the second preset field, the user attribute data and the first user identifiers corresponding to the user identifiers.

[0220] In some embodiments of the present application, the data processing apparatus further comprises an accumulation module configured to accumulate the first user identifiers to obtain an accumulated number; a first user identifier acquisition module configured to acquire, if the accumulated number reaches a set number threshold, a first user identifier corresponding to a user with the longest inactive duration; and an assignment module configured to assign the first user identifier corresponding to the user with the longest inactive duration to a user to be generated with a next first user identifier.

[0221] In some embodiments of the present application, the efficient compression bitmap data table further comprises a user group data table; the data processing apparatus comprises a user identifier set determination module configured to determine a user identifier set corresponding to a user group according to user identifiers corresponding to users in the user group; a first user identifier set determination module configured to determine first user identifiers corresponding to user identifiers in the user identifier set according to a mapping relationship between the user identifiers and the first user identifiers, to obtain a first user identifier set corresponding to the user group; a compression module configured to perform efficient bitmap compression on the first user identifiers in the first user identifier set to obtain an efficient compression bitmap corresponding to the user group; and a user group data table storage module configured to store the efficient compression bitmap corresponding to the user group and a user group identifier corresponding to the user group in the user group data table.

[0222] In some embodiments of the present application, the query request comprises at least one of an event analysis request, a retention analysis request, a funnel analysis request and a user path analysis request.

[0223] Figure 12 A structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown.

[0224] It should be noted that, Figure 12 The computer system 1200 of the electronic device shown is only an example and should not impose any limitation on the functions and use range of embodiments of the present application.

[0225] As ​As shown, the computer system 1200 includes a central processing unit (CPU) 1201 that can perform various suitable actions and processes in accordance with programs stored in a read-only memory (ROM) 1202 or loaded into a random access memory (RAM) 1203 from a storage section 1208, such as performing the methods in the above-described embodiments. Various programs and data required for the operation of the system are also stored in the RAM 1203. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0226] Connected to the I / O interface 1205 are an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as necessary. A removable recording medium 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1210 as necessary, so that a computer program read therefrom is installed into the storage section 1208 as necessary.

[0227] In particular, in accordance with the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 1209, and / or installed from the removable recording medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are performed.

[0228] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, transmit, propagate or transport a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0229] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment, or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0230] The units described in the embodiments of the present application can be implemented by software, or can be implemented by hardware, and the units described can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0231] As another aspect, the present application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable storage medium carries computer readable instructions, which, when executed by a processor, implement the method in any of the above embodiments.

[0232] According to an aspect of the present application, an electronic device is also provided, which includes a processor, and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method in any of the above embodiments.

[0233] According to an aspect of the embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method in any of the above embodiments.

[0234] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into a plurality of modules or units.

[0235] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or network, and includes a plurality of instructions to make a computing device (which can be a personal computer, server, touch terminal, or network device, etc.) execute the method according to the embodiments of the present application.

[0236] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the application embrace any and all variations of the present application that fall within the scope of the general inventive concept as defined by the appended claims and their equivalents. It is intended that the application encompass all such variations as fall within the scope of the appended claims and their equivalents.

[0237] It is to be understood that the application is not limited to the precise construction herein described and as shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should only be limited by the claims appended hereto.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a query request, the query request indicating query information, the query information comprising condition limiting information; parsing the condition limiting information to determine at least two query conditions corresponding to the condition limiting information and a Boolean logical relationship between the at least two query conditions; generating a query code corresponding to each of the at least two query conditions respectively; the query code is used to query an object in a high-efficiency compressed bitmap data table to obtain a first high-efficiency compressed bitmap of an object set satisfying the corresponding query condition, the high-efficiency compressed bitmap data table comprising high-efficiency compressed bitmaps of a plurality of initial object sets; the initial object set is determined by classifying objects according to attribute values; combining the query codes corresponding to the query conditions according to the Boolean logical relationship between the at least two query conditions to obtain a combined code, comprising: obtaining a high-efficiency compressed bitmap function corresponding to the Boolean logical relationship according to the Boolean logical relationship between the at least two query conditions; combining the query codes corresponding to the query conditions according to the high-efficiency compressed bitmap function corresponding to the Boolean logical relationship to obtain the combined code; the combined code is used to perform logical operation on the first high-efficiency compressed bitmap according to the Boolean logical relationship between the at least two query conditions to obtain a target high-efficiency compressed bitmap; and the target high-efficiency compressed bitmap is used to determine a query result.

2. The method of claim 1, wherein, The query information further comprises classification statistical information, the classification statistical information indicating target information items that need to be counted according to attribute values; The method further comprises: generating a classification statistical indication code according to the classification statistical information; after the combining code is obtained by combining the query codes corresponding to the query conditions according to the Boolean logical relationship between the at least two query conditions, the method further comprises: updating the combined code according to the classification statistical indication code, the updated combined code being used to classify an object set indicated by the target high-efficiency compressed bitmap according to attribute values of the target information items.

3. The method according to claim 1 or 2, characterized in that, The query request comprises a query task identifier, the query task identifier being generated by an initiator of the query request upon detecting an input operation on the query information; after the query request is obtained, the method further comprises: obtaining the query information associated with the query task identifier from a specified information table; the initiator of the query request stores the query task identifier and the detected query information in the specified information table in association after detecting the input operation on the query information; after the combining code is obtained by combining the query codes corresponding to the query conditions according to the Boolean logical relationship between the at least two query conditions, the method further comprises: storing the combined code corresponding to the query information and the query task identifier in the specified information table in association; and storing the combined code corresponding to the query information and the query task identifier in the specified information table in association. return deposit indication information to an initiator of the query request, so that the initiator of the query request acquires and executes the combination code from the specified information table according to the deposit indication information.

4. The method according to claim 1 or 2, characterized in that, The object is a user; Before the generating the query code corresponding to each of the at least two query conditions respectively, the method further comprises: obtaining user operation data and user attribute data, the user operation data being used to indicate an interactive behavior of a user on a user interface of a product; performing efficient bitmap compression on the first user identifiers corresponding to the same attribute values according to the attribute values of the fields in the user operation data and the attribute values of the fields in the user attribute data, to obtain efficient compressed bitmaps corresponding to the attribute values; storing the obtained efficient compressed bitmaps and the corresponding attribute values in the efficient compressed bitmap data table.

5. The method of claim 4, wherein, The efficient compressed bitmap data table comprises a second data table and a third data table; The performing efficient bitmap compression on the first user identifiers corresponding to the same attribute values according to the attribute values of the fields in the user operation data and the attribute values of the fields in the user attribute data, to obtain efficient compressed bitmaps corresponding to the attribute values, comprises: generating a user operation table according to the user operation data, and generating a user attribute table according to the user attribute data, the user operation table and the user attribute table comprising the first user identifiers corresponding to each user; performing efficient bitmap compression on the first user identifiers corresponding to the same attribute values in the user operation table according to the attribute values of the fields in the user operation table, to obtain second efficient compressed bitmaps associated with the corresponding fields and the corresponding attribute values; and performing efficient bitmap compression on the first user identifiers corresponding to the same attribute values in the user attribute table according to the attribute values of the fields in the user attribute table, to obtain third efficient compressed bitmaps associated with the corresponding fields and the corresponding attribute values. The storing the obtained efficient compressed bitmaps and the corresponding attribute values in the efficient compressed bitmap data table, comprises: storing the second efficient compressed bitmaps and the corresponding fields and the corresponding attribute values in the second data table; storing the third efficient compressed bitmaps and the corresponding fields and the corresponding attribute values in the third data table.

6. The method of claim 4, wherein, The user operation data and the user attribute data comprise user identifiers; Before the generating a user operation table according to the user operation data, and generating a user attribute table according to the user attribute data, the method further comprises: if the user identifiers are in a format that does not support efficient bitmap compression, generating the first user identifiers corresponding to each user identifier in a specified format that supports efficient bitmap compression; The generating a user operation table according to the user operation data, and generating a user attribute table according to the user attribute data, comprises: generating the user operation table according to a first preset field, the user operation data and the first user identifiers corresponding to each user identifier; and generating the user attribute table according to a second preset field, the user attribute data and the first user identifiers corresponding to each user identifier.

7. The method of claim 6, wherein, The method further comprises: The first user identifier is accumulated to obtain an accumulated number; If the accumulated number reaches a set number threshold, a first user identifier corresponding to a user with the longest inactive duration is obtained; The first user identifier corresponding to the user with the longest inactive duration is assigned to a next user to be generated with a first user identifier.

8. The method of claim 6, wherein, The high-efficiency compressed bitmap data table further includes a user group data table; and the method further includes: According to user identifiers corresponding to users in a user group, a user identifier set corresponding to the user group is determined; According to a mapping relationship between the user identifiers and the first user identifiers, first user identifiers corresponding to user identifiers in the user identifier set are determined, to obtain a first user identifier set corresponding to the user group; The first user identifiers in the first user identifier set are compressed by high efficiency to obtain a high-efficiency compressed bitmap corresponding to the user group; The high-efficiency compressed bitmap corresponding to the user group and a user group identifier corresponding to the user group are stored in the user group data table.

9. The method of claim 4, wherein, The query request includes at least one of an event analysis request, a retention analysis request, a funnel analysis request, and a user path analysis request.

10. A data processing apparatus, characterized by, The method includes: a query request obtaining module configured to obtain a query request, the query request indicating query information, the query information including condition limiting information; a parsing module configured to parse the condition limiting information to determine at least two query conditions corresponding to the condition limiting information and a Boolean logic relationship between the at least two query conditions; a query code generating module configured to generate a query code corresponding to each of the at least two query conditions respectively; the query code is used to query an object in a high-efficiency compressed bitmap data table to obtain a first high-efficiency compressed bitmap of an object set satisfying a corresponding query condition, the high-efficiency compressed bitmap data table including high-efficiency compressed bitmaps of a plurality of initial object sets; the initial object set is determined according to classification of objects according to attribute values; a combination module configured to combine the query codes corresponding to the at least two query conditions according to the Boolean logic relationship between the at least two query conditions to obtain a combination code, including: obtaining a high-efficiency compressed bitmap function corresponding to the Boolean logic relationship according to the Boolean logic relationship between the at least two query conditions; and combining the query codes corresponding to the at least two query conditions according to the high-efficiency compressed bitmap function corresponding to the Boolean logic relationship to obtain the combination code; the combination code is used to perform a logical operation on the first high-efficiency compressed bitmap according to the Boolean logic relationship between the at least two query conditions to obtain a target high-efficiency compressed bitmap; and the target high-efficiency compressed bitmap is used to determine a query result.

11. The apparatus of claim 10, wherein, The query information further includes classification statistical information, the classification statistical information indicating target information items that need to be counted according to attribute values; The data processing apparatus further includes: a classification statistical indication code generating module configured to generate a classification statistical indication code according to the classification statistical information; An updating module is configured to update the combination code according to the classification statistical indication code, and the updated combination code is used to classify the object set indicated by the target high-efficiency compressed bitmap according to the attribute value of the target information item.

12. The apparatus of claim 10 or 11, wherein, The query request includes a query task identifier, which is generated by an initiator of the query request when detecting an input operation for query information; The data processing apparatus further includes: A query information obtaining module is configured to obtain the query information associated with the query task identifier from a designated information table; the initiator of the query request stores the query task identifier and the detected query information in the designated information table in association after detecting the input operation for query information; A combination code storing module is configured to store the combination code corresponding to the query information in the designated information table in association with the query task identifier; A storing indication information returning module is configured to return storing indication information to the initiator of the query request, so that the initiator of the query request obtains and executes the combination code from the designated information table according to the storing indication information.

13. The apparatus of claim 10 or 11, wherein, The object is a user; the data processing apparatus further includes: A data obtaining module is configured to obtain user operation data and user attribute data, the user operation data being used to indicate the interactive behavior of the user on the user interface of the product; An efficient bitmap compression module is configured to compress the first user identifiers corresponding to the same attribute value in the user operation data and the user attribute data according to the attribute values of the fields in the user operation data and the attribute values of the fields in the user attribute data, to obtain the efficient compressed bitmap corresponding to the attribute value. A storage module is configured to store the obtained efficient compressed bitmap and the corresponding attribute value in the efficient compressed bitmap data table.

14. The apparatus of claim 13, wherein, The efficient compressed bitmap data table includes a second data table and a third data table; the efficient bitmap compression module includes: A generating unit is configured to generate a user operation table according to the user operation data and a user attribute table according to the user attribute data, the user operation table and the user attribute table including the first user identifiers corresponding to each user; A second efficient compressed bitmap generating unit is configured to compress the first user identifiers corresponding to the same attribute value in the user operation table according to the attribute values of the fields in the user operation table, to obtain the second efficient compressed bitmap associated with the corresponding field and the corresponding attribute value; and A third efficient compressed bitmap generating unit is configured to compress the first user identifiers corresponding to the same attribute value in the user attribute table according to the attribute values of the fields in the user attribute table, to obtain the third efficient compressed bitmap associated with the corresponding field and the corresponding attribute value. The storage module includes: A first storage unit is configured to store the second efficient compressed bitmap, the corresponding field and the corresponding attribute value in the second data table; A second storage unit is configured to store the third efficient compressed bitmap, the corresponding field and the corresponding attribute value in the third data table.

15. The apparatus of claim 13, wherein, The user operation data and the user attribute data comprise user identifiers; and the data processing apparatus further comprises: a first user identifier generation unit configured to, if the user identifiers are in a format that does not support efficient bitmap compression, generate first user identifiers corresponding to the user identifiers in a specified format that supports efficient bitmap compression; the generation unit comprises: a user operation table generation unit configured to generate the user operation table according to the first preset field, the user operation data, and the first user identifiers corresponding to the user identifiers; and a user attribute table generation unit configured to generate the user attribute table according to the second preset field, the user attribute data, and the first user identifiers corresponding to the user identifiers.

16. The apparatus of claim 15, wherein, The data processing apparatus further comprises: an accumulation module configured to accumulate the first user identifiers to obtain an accumulated number; a first user identifier acquisition module configured to, if the accumulated number reaches a set number threshold, acquire a first user identifier corresponding to a user with the longest inactive duration; an allocation module configured to allocate the first user identifier corresponding to the user with the longest inactive duration to a user for which a next first user identifier is to be generated.

17. The apparatus of claim 15, wherein, The efficient compression bitmap data table further comprises a user group data table; and the data processing apparatus further comprises: a user identifier set determination module configured to determine a user identifier set corresponding to a user group according to user identifiers corresponding to users in the user group; a first user identifier set determination module configured to determine first user identifiers corresponding to user identifiers in the user identifier set according to a mapping relationship between the user identifiers and the first user identifiers, to obtain a first user identifier set corresponding to the user group; a compression module configured to perform efficient bitmap compression on the first user identifiers in the first user identifier set to obtain an efficient compression bitmap corresponding to the user group; a user group data table storage module configured to store the efficient compression bitmap corresponding to the user group and a user group identifier corresponding to the user group in the user group data table.

18. The apparatus of claim 13, wherein, The query request comprises at least one of an event analysis request, a retention analysis request, a funnel analysis request, and a user path analysis request.

19. An electronic device, comprising: comprises: a processor; a memory having computer readable instructions stored thereon, the computer readable instructions being executable by the processor to implement the method of any one of claims 1-9.

20. A computer readable storage medium having computer readable instructions stored thereon, the computer readable instructions being executable by a processor to implement the method of any one of claims 1-9.

21. A computer program product, characterised in that, comprise computer instructions executable by a processor to implement the method of any one of claims 1-9.

Images