Data processing method and device and computing equipment

Abstract

The invention discloses a data processing method and device and computing equipment. The method comprises the following steps: receiving a creation request for a target object set sent by a client; generating a single-label conditional statement and a multi-label aggregation statement corresponding to the creation request; executing the single-tag conditional statement to generate a single-tag result bitmap; executing the multi-label aggregation statement to perform bitmap operation on the single-label result bitmap to obtain an intermediate result bitmap, and performing bitmap operation on the intermediate result bitmap and / or the single-label result bitmap to generate a target object set bitmap. According to the scheme, storage resources can be greatly saved through bitmap storage, the object set creation efficiency can be greatly improved through fusion of the single-label conditional statement, the multi-label aggregation statement and bitmap operation, and the data processing speed is increased.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and specifically to a data processing method, apparatus, computing device, computer storage medium, and computer program product. Background Art

[0002] With the continuous development of science and technology and society, the emergence of various platforms has greatly enriched people's work and life. In order to facilitate information recommendation and improve user experience, some platforms usually create different object sets, each of which is a group of objects with specific common attributes, characteristics or behaviors.

[0003] However, the inventors discovered during implementation that the object set creation efficiency in the prior art is low, especially when the amount of data is large, the object set creation takes a long time and cannot meet the demand for efficient creation of object sets. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a data processing method, apparatus, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of the present application, a data processing method is provided, comprising:

[0006] Receive a creation request for a target object collection sent by a client;

[0007] Generate a single-label conditional statement and a multi-label aggregation statement corresponding to the creation request;

[0008] Executing the single-label conditional statement to generate a single-label result bitmap;

[0009] The multi-label aggregation statement is executed to perform a bitmap operation on the single-label result bitmap to obtain an intermediate result bitmap, and a bitmap operation is performed on the intermediate result bitmap and / or the single-label result bitmap to generate a target object set bitmap.

[0010] In an optional implementation, generating a single-label conditional statement and a multi-label aggregation statement corresponding to the creation request includes:

[0011] Parsing the creation request to determine target tags associated with the target object set, constraints of the target tags, and logical relationships between the target tags;

[0012] Querying the tag meta information matching the target tag and the constraint condition from a pre-stored tag meta information table;

[0013] For each target tag, a single-tag conditional statement of the target tag is generated according to the constraint conditions of the target tag, tag meta information and bitmap generation function;

[0014] Determine a bitmap operation function that matches the logical relationship, and generate the multi-label aggregation statement according to the bitmap operation function.

[0015] In an optional embodiment, the target tag includes: an object tag and / or a collection tag;

[0016] Then the generating of the single label conditional statement corresponding to the creation request includes: for the object label, generating the object label conditional statement corresponding to the object label; and / or, for the set label, generating the set label conditional statement corresponding to the object label;

[0017] Then executing the single-label conditional statement to generate a single-label result bitmap includes: based on a pre-stored object label data table, executing the object label conditional statement to generate a single-label result bitmap; and / or, based on a pre-stored object collection bitmap, executing the collection label conditional statement to generate a single-label result bitmap.

[0018] In an optional embodiment, the method further includes:

[0019] Get initial label data;

[0020] Adding tag data corresponding to the same tag type to the same object tag data table; and / or, adding tag data corresponding to the same operation frequency range to the same object tag data table;

[0021] The number of labels contained in the object label data table is less than a preset number threshold.

[0022] In an optional implementation, the method further includes: generating a data table index of the object tag data table; wherein the data table index is sorted in a preset order.

[0023] In an optional implementation, the preset order is: the tag priority of the object tags included in the object tag data table.

[0024] In an optional embodiment, the method further includes:

[0025] Count the operation frequencies of different object tags;

[0026] The tag priority of the object tag is determined according to the operation frequency.

[0027] According to a second aspect of the present application, a data processing device is provided, including:

[0028] A receiving module, used for receiving a creation request for a target object set sent by a client;

[0029] A generation module, used to generate a single-label conditional statement and a multi-label aggregation statement corresponding to the creation request;

[0030] An execution module is used to execute the single-label conditional statement to generate a single-label result bitmap; execute the multi-label aggregation statement to perform a bitmap operation on the single-label result bitmap to obtain an intermediate result bitmap, and perform a bitmap operation on the intermediate result bitmap and / or the single-label result bitmap to generate a target object set bitmap.

[0031] In an optional implementation, the generation module is used to: parse the creation request, determine the target tags associated with the target object set, the constraints of the target tags, and the logical relationship between the target tags;

[0032] Querying the tag meta information matching the target tag and the constraint condition from a pre-stored tag meta information table;

[0033] For each target tag, a single-tag conditional statement of the target tag is generated according to the constraint conditions of the target tag, tag meta information and bitmap generation function;

[0034] Determine a bitmap operation function that matches the logical relationship, and generate the multi-label aggregation statement according to the bitmap operation function.

[0035] In an optional embodiment, the target tag includes: an object tag and / or a collection tag;

[0036] The generating module is used to: generate, for an object tag, an object tag conditional statement corresponding to the object tag; and / or, for a set tag, generate a set tag conditional statement corresponding to the object tag;

[0037] The execution module is used to: execute the object label conditional statement to generate a single label result bitmap based on a pre-stored object label data table; and / or execute the set label conditional statement to generate a single label result bitmap based on a pre-stored object set bitmap.

[0038] In an optional embodiment, the device also includes: a label data storage module for acquiring initial label data; adding label data corresponding to the same label type to the same object label data table; and / or adding label data corresponding to the same operation frequency range to the same object label data table; wherein the number of labels contained in the object label data table is less than a preset number threshold.

[0039] In an optional implementation, the label storage module is used to: generate a data table index of the object label data table; wherein the data table index is sorted in a preset order.

[0040] In an optional implementation, the preset order is: the tag priority of the object tags included in the object tag data table.

[0041] In an optional implementation, the tag priority is obtained in the following manner: counting the operation frequencies of different object tags; and determining the tag priority of the object tag according to the operation frequencies.

[0042] According to a third aspect of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0043] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned data processing method.

[0044] According to a fourth aspect of the present application, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned data processing method.

[0045] According to a fifth aspect of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned data processing method.

[0046] The embodiment of the present application converts the creation request for the target object set sent by the client into a single-label conditional statement and a multi-label aggregation statement corresponding to each label. The single-label conditional statement can convert the processing result for a single label into bitmap storage, and the multi-label aggregation statement performs bitmap operation based on the single-label result bitmap of the single-label conditional statement to obtain an intermediate result bitmap, and then obtains the target object set bitmap. Therefore, storage resources can be greatly saved through bitmap storage, and the integration of single-label conditional statements and multi-label aggregation statements with bitmap operations can greatly improve the efficiency of object set creation and increase data processing speed.

[0047] The embodiment of the present application generates, for each target tag, a single-label conditional statement of the target tag based on the constraint conditions, tag meta-information and bitmap generation function of the target tag; and generates a multi-label aggregation statement based on the bitmap operation function corresponding to the logical relationship between the target tags, thereby improving the generation accuracy of the operation statement corresponding to the creation request and ensuring data processing accuracy.

[0048] The tags in the embodiment of the present application include object tags and collection tags, so that the user can circle a new collection by object tags, and can also circle a new collection based on object tags and generated collections.

[0049] The embodiment of the present application ensures that the number of tags in the object tag data table is lower than a certain threshold, thereby improving the scalability of the data table and improving data query efficiency.

[0050] The embodiment of the present application adds the tag data corresponding to the same tag type to the same object tag data table, and the tags contained in the generated object tag data table correspond to the same tag type, thereby reducing the number of tags contained in a single object tag data table. And when executing a single-tag conditional statement later, the data table where the target tag is located can be quickly located by the tag type in the tag meta information, thereby improving the overall execution efficiency.

[0051] The embodiment of the present application adds tag data corresponding to the same operation frequency range to the same object tag data table, thereby making the tag operation frequency in the same object tag data table consistent, reducing cache data jitter, and facilitating maintenance and management of the data table.

[0052] The object tag data table in the embodiment of the present application is indexed and sorted according to the tag priority of the object tags contained in the object tag data table, which is beneficial to improving the execution efficiency of single-tag conditional statements and thus improving the overall processing efficiency.

[0053] The tag priority in the embodiment of the present application is determined according to the tag operation frequency, so that the tags with high operation frequency can be indexed and sorted first, which is beneficial to improving data processing efficiency.

[0054] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0056] Figure 1 A schematic diagram of an operating environment provided for implementing at least one embodiment of the present application is shown;

[0057] Figure 2A flow chart of a data processing method provided in Embodiment 1 of the present application is shown;

[0058] Figure 3 A flowchart of a sentence generation method provided in Embodiment 1 of the present application is shown;

[0059] Figure 4 A flow chart of a data processing method provided in Embodiment 2 of the present application is shown;

[0060] Figure 5 A schematic diagram of a client collection selection interface provided in Embodiment 2 of the present application is shown;

[0061] Figure 6 A schematic diagram of the structure of a data processing device provided in Embodiment 3 of the present application is shown;

[0062] Figure 7 A schematic diagram of the structure of a computing device provided in Embodiment 4 of the present application is shown. DETAILED DESCRIPTION

[0063] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0064] It should be noted that the information and data of the objects involved in the embodiments of the present application (such information and data include but are not limited to information and data used for analysis, information and data stored, information and data displayed, etc.) are all authorized or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for selection of authorization or rejection.

[0065] Figure 1 The present invention can be applied to an application environment including, but not limited to, a client 2 , a server 4 , and a network 6 .

[0066] in:

[0067] The server 4 may be composed of a single or multiple computing devices. The multiple computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device may load a virtual machine based on a virtual image and / or other data defining specific software (e.g., operating system, dedicated application, server) for simulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices. A hypervisor may be implemented to manage the use of different virtual machines on the same computing device.

[0068] The server 4 may be configured to communicate with the client 2, etc., via a network 6. The network 6 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 6 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, etc., or wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0069] The server 4 can provide storage, reading, downloading, writing, querying, deleting and other services, such as providing static resource download services to clients through multiple domain names.

[0070] Client 2 can be running Windows, Android TM ) or IOS and other operating systems, such as smart phones, tablet devices, laptop computers, virtual reality devices, gaming devices, set-top boxes, car terminals, smart TVs. Based on the above operating systems, various applications, such as browsers, can be run.

[0071] Embodiment 1

[0072] Figure 2 A flow chart of a data processing method provided in the first embodiment of the present application is shown. The data processing method provided in the embodiment of the present application can be executed on the server side.

[0073] Specifically, Figure 2 As shown, the method comprises the following steps:

[0074] Step S210: receiving a creation request for a target object set sent by a client.

[0075] In the actual implementation process, the client needs to select a set of objects with one or more identical features, such as a user set, for corresponding business needs, such as building a precision marketing project or a user classification management system. Then, a set creation request can be initiated through the corresponding operation of the client. The set that the set creation request expects to create is the target object set. The server receives the creation request for the target object set sent by the client.

[0076] Step S220, generating a single-label conditional statement and a multi-label aggregation statement corresponding to the creation request.

[0077] A label is a highly refined feature of an object that is derived from the object's own attributes or by analyzing and calculating the object's information. The labels in the embodiments of the present application may include object labels and collection labels. An object label refers to a label assigned to an object based on the original data associated with the object (such as original log data, behavior data, attribute data, etc.), such as an age label, a preference label, etc.; a collection label refers to a collection label assigned to an object through the creation of an object collection, which is used to characterize the collection to which the object belongs.

[0078] The creation request for the target object set received by the server carries the relevant constraints of the target object set. By parsing the creation request, the corresponding operation statement, such as the corresponding SQL statement, can be obtained.

[0079] The prior art generates operation statements based on data tables. For example, the operation statement generated by the prior art is usually “SELECT name, age FROM xx.table”, that is, the data table corresponds to the operation statement one by one.

[0080] This application is different from the statement generation method in the prior art. This application generates operation statements based on labels, and finally generates single-label conditional statements and multi-label aggregation statements. Among them, the single-label conditional statement only involves the operation of a single label data, and the multi-label aggregation statement is the aggregation of the operation results of multiple label data. The multi-label aggregation statement does not involve the processing of the internal data of a single label.

[0081] In an optional implementation, it can be specifically adopted Figure 3 The steps shown are used to generate the operation statement corresponding to the creation request, thereby improving the generation accuracy of the operation statement:

[0082] S221, parsing the creation request, determining the target tags associated with the target object set, the constraints of the target tags, and the logical relationship between the target tags.

[0083] The creation request carries the condition information for creating the target object set, which contains the label features that the target object set should have. The condition information carries a label identifier, and the label corresponding to the label identifier is the target label. Moreover, the constraints of each target label and the logical relationship between different target labels are extracted from the condition information.

[0084] S222: Query the tag meta information that matches the target tag and the constraint condition from the pre-stored tag meta information table.

[0085] The server pre-generates and stores a tag meta information table, which records the meta information of each tag. The meta information includes but is not limited to: tag ID, tag name, tag type (such as category name and / or category ID under different levels of classification), tag data table, tag association table field, optional value range, data type, creation time, modification time view ID, etc.

[0086] For each target tag, the tag meta information related to the target tag, such as the data table where the target tag is located, the associated fields, and the data type, is obtained from the tag meta information table.

[0087] S223: For each target tag, generate a single tag conditional statement of the target tag according to the constraint conditions of the target tag, tag meta information and a bitmap generation function.

[0088] The target tag constraint and tag meta information can determine the operation table, operation field, tag operator and operand of the single tag conditional statement, and then generate the corresponding data operation statement. The tag operator includes but is not limited to: >=, <=, >, <, =, BETWEEN AND, IN, NOT IN, IS NOT NULL, IS NULL; the bitmap generation function can convert the result of the data operation statement into a bitmap.

[0089] Wherein, the target tag in the embodiment of the present application includes an object tag and / or a collection tag, then for the object tag, an object tag conditional statement corresponding to the object tag is generated, and / or, for the collection tag, a collection tag conditional statement corresponding to the object tag is generated. Wherein, the object tag conditional statement is for the object tag data table, and the collection tag conditional statement is for the object collection bitmap; and there are also differences in the operators of the two, the operators in the object tag conditional statement may include >=, <=, >, <; while the operators in the collection tag conditional statement do not include >=, <=, >, <, etc.

[0090] Among them, the object label conditional statement can be as follows:

[0091]

[0092] A collection tag conditional statement can look like this:

[0093]

[0094]

[0095] Among them, groupBitmapState is the bitmap generation function; "dws_op_dmp_label_uid_game_base_id_narrow_a_d" is the object label data table; "dws_op_dmp_crowd_uid_bit_map_a_d" is the object collection bitmap.

[0096] S224, determining a bitmap operation function that matches the logical relationship, and generating a multi-label aggregation statement according to the bitmap operation function.

[0097] The bitmap operation function is used to process the logical relationship between target tags, thereby generating a multi-tag aggregation statement. Specifically, the multi-tag aggregation statement is obtained according to the identifier of the single-tag result bitmap in the single-tag conditional statement and the bitmap operation function.

[0098] For example, the generated multi-label aggregation statement can be as follows:

[0099] bitmapAndnot(bitmapAnd(innerTable,unionTable),antiTable)

[0100] Among them, bitmapAnd represents the bitmap intersection operation function; bitmapAndnot represents the bitmap difference operation function; innerTable, unionTable, and antiTable represent the execution results of single-label statements.

[0101] Step S230, executing the single-label conditional statement to generate a single-label result bitmap.

[0102] Each single-label conditional statement is executed, and the execution result of each single-label conditional statement is stored in a bitmap, so that each single-label conditional statement obtains a corresponding single-label result bitmap.

[0103] Specifically, based on a pre-stored object label data table, an object label conditional statement is executed to generate a single label result bitmap, that is, the object label conditional statement is executed according to the object label data table; and / or, based on a pre-stored object collection bitmap, a collection label conditional statement is executed to generate a single label result bitmap, that is, the collection label conditional statement is generated according to the object collection bitmap.

[0104] Step S240, executing the multi-label aggregation statement to perform bitmap operation on the single-label result bitmap to obtain an intermediate result bitmap, and performing bitmap operation on the intermediate result bitmap and / or the single-label result bitmap to generate a target object set bitmap.

[0105] Parse and generate a syntax tree of a multi-label aggregated statement, perform bitmap operations layer by layer according to the syntax tree using a bitmap operation function, store the intermediate results obtained by each bitmap operation in a bitmap, and thus obtain an intermediate result bitmap; finally, perform a bitmap operation on the intermediate result bitmap and / or the single-label result bitmap, obtain a result as a target object set result, and store the target object set result in a bitmap, that is, obtain a target object set bitmap. Among them, each bitmap generated in the embodiment of the present application can be stored in a bitmap compression storage method, thereby greatly saving storage resources.

[0106] It can be seen that the data processing method provided by the embodiment of the present application converts the creation request for the target object set sent by the client into a single-label conditional statement and a multi-label aggregation statement corresponding to each label. The single-label conditional statement can convert the processing result for a single label into bitmap storage, and the multi-label aggregation statement performs a bitmap operation based on the single-label result bitmap of the single-label conditional statement to obtain an intermediate result bitmap, and then obtains the target object set bitmap. Therefore, storage resources can be greatly saved through bitmap storage, and the integration of single-label conditional statements and multi-label aggregation statements with bitmap operations can greatly improve the efficiency of object set creation and improve data processing speed.

[0107] Embodiment 2

[0108] Figure 4 A flow chart of a data processing method provided in the second embodiment of the present application is shown. The data processing method provided in the embodiment of the present application is executed in a data processing system.

[0109] Specifically, Figure 4 As shown, the method comprises the following steps:

[0110] Step S410, synchronizing data tags.

[0111] The data warehouse stores a large amount of object data, such as object attribute information, behavior data, etc. The corresponding initial label data can be obtained by cleaning the object data and processing it through machine learning models and preset label assignment algorithms.

[0112] The prior art usually uses a wide table to record all the tag data of an object, which is inefficient. The embodiment of the present application obtains initial tag data and generates multiple narrow tables based on the initial tag data, that is, the number of tags contained in the generated object tag data table is less than the preset number threshold. This improves the scalability of the data table, realizes on-demand storage, and improves data query efficiency.

[0113] The specific method for generating the object tag data table may be a combination of one or more of the following implementations.

[0114] Implementation method one: add tag data corresponding to the same tag type to the same object tag data table. Specifically, the tags are divided into multiple types in advance, for example, they can be divided into game types, user types, platform types, game meta types, etc. Then add the tag data belonging to the same tag type to the same object tag data table, and the tags contained in the object tag data table thus generated correspond to the same tag type, thereby reducing the number of tags contained in a single data table. And when executing a single-tag conditional statement later, the data table where the target tag is located can be quickly located by the tag type in the tag meta information, thereby improving the overall execution efficiency.

[0115] Implementation method two: Add the tag data corresponding to the same operation frequency range to the same object tag data table. Specifically, the operation frequency for each tag in the server is counted in advance, and different operation frequency ranges are set, and then the tag data corresponding to the same operation frequency range is added to the same object tag data table. For example, the tag data of the object tag whose operation frequency is between {N1, N2} is added to an object tag data table. The use of this implementation method can make the tag operation frequency in the same object tag data table consistent, reduce cache data jitter, and facilitate the maintenance and management of the data table.

[0116] The data warehouse can synchronize the generated object tag data table to the server, and can also synchronize the initial tag data to the server.

[0117] Step S420, storing the object tag data table.

[0118] If the data warehouse synchronizes the initial tag data to the server, the server executes the steps of adding the tag data corresponding to the same tag type to the same object tag data table; and / or, adding the tag data corresponding to the same operation frequency range to the same object tag data table, thereby generating and storing the object tag data table; if the data warehouse synchronizes the object tag data table to the server, the object tag data table is directly stored. In the storage process, the object tag data table can be partitioned to improve the query speed.

[0119] In an optional implementation, in order to improve data query efficiency, the embodiment of the present application generates a data table index of the object tag data table, and the data table index is sorted in a preset order, that is, the data table index in the embodiment of the present application is an ordered index.

[0120] Further optionally, the preset order may be determined according to the tag priority of the object tag included in the object tag data table. Specifically, for each object tag included in the object tag data table, the tag priority of each object tag is determined, and the tags are sorted in descending order according to the tag priority. The tag value of the object tag with the highest tag priority may be indexed and sorted. For example, if the priority of object tag 1 is higher than that of object tag 2, the tag value of object tag 1 is preferentially indexed and sorted. If the tag values ​​of object tag 1 are the same, the tag value of object tag 2 is indexed and sorted.

[0121] Optionally, the operation frequency of different object tags can be counted, and the tag priority of the object tag can be determined based on the operation frequency. That is, the higher the operation frequency, the higher the tag priority of the object tag. In this way, the tag values ​​of object tags with high operation frequency can be prioritized for sorting, so that when the single-tag conditional statement is executed later, the execution result can be obtained quickly, improving the overall execution efficiency.

[0122] Step S430, executing a collection selection operation.

[0123] The client provides a visual interface, and the client user can perform the corresponding collection selection operation. Specifically, the embodiment of the present application can provide a collection selection interface based on tags on the client. Figure 5 As shown, an object label selection area is provided on the left side of the interface, in which labels previously selected by the user are displayed, as well as different label types, and after a label type is triggered, each object label belonging to the label type is displayed; a label operation area is provided in the middle of the interface, and the user can add labels to different logical relationship areas (such as intersection areas, union areas, etc.) by dragging and other operations, and can enter constraint conditions for the selected object labels in the conditional input ports in the corresponding areas, such as value ranges, etc.; an effective period operation entrance is provided on the right side of the interface, and the user can configure the effective period of the selected set through the effective period operation entrance, and the set will be automatically cleared after the effective period expires; a quantity estimate is also provided on the right side of the interface, and the estimate and display of the number of objects contained in the selected set can be triggered through this entrance.

[0124] In addition to performing collection selection operations, the client can also perform tag management operations, collection viewing operations, etc. For example, the client user can delete, add, and modify the corresponding object tag, and then modify the corresponding object tag data table accordingly; after the client performs a collection viewing operation, the server can convert the corresponding collection bitmap into an array and feed it back to the client.

[0125] Step S440: Send a creation request for the target object set.

[0126] The client generates a create request for the target object collection based on the collection selection operation and sends it to the server.

[0127] Step S450: Generate an operation statement corresponding to the creation request.

[0128] The server generates an operation statement corresponding to the creation request, which includes a single-label conditional statement and a multi-label aggregation statement.

[0129] Step S460: execute the operation statement to generate a target object set bitmap.

[0130] Specifically, before executing an operation statement, determine whether a statement-collection bitmap record matching the operation statement is currently stored, and the statement-collection bitmap record stores object collection bitmap information obtained after executing operation statements corresponding to different requests. If there is currently a statement-collection bitmap record matching the current operation statement, indicating that the corresponding object collection bitmap has been generated for the creation request, the collection bitmap corresponding to the operation statement in the collection bitmap record is used as the target object collection bitmap; if there is currently no statement-collection bitmap record matching the current operation statement, the operation statement is executed. In addition, the index of the target object collection bitmap can be arranged in a preset order to improve execution efficiency. Among them, the generation process and execution process of the operation statement can refer to the description in other embodiments, which will not be repeated here.

[0131] Step S470: storing the target object set bitmap and quantity statistics and / or bitmap conversion.

[0132] The generated target object set bitmap is stored, and the mapping relationship between the operation statement and the target object set bitmap is cached, thereby generating a statement-set bitmap record. When the cache time of the statement-set bitmap record reaches a preset time, and / or the object tag data table associated with the statement is updated, the statement-set bitmap record is deleted.

[0133] In addition, the number of objects included in the target object set may be counted, and / or the target object set bitmap may be converted into an array, etc., such as converting the bitmap into a corresponding array through a bitmapToArray function.

[0134] In addition, in order to improve data stability, the task processing progress or task processing status is detected in real time. When the time interval between the current time and the time when the client initiates the creation request reaches the preset time length, if the target object set bitmap has not been generated, S450 and subsequent steps are re-executed. And when the number of retries reaches the preset number threshold, or when a delay or abnormality occurs during the generation of the target object set bitmap, a corresponding alarm message is generated and fed back to the processing end.

[0135] Step S480, return the result.

[0136] The server returns the quantity statistics results, etc. for visual display in the client.

[0137] In an optional implementation, the data processing method provided in the embodiment of the present application may adopt the following architecture: the architecture includes a data source layer, an input layer, a processing layer, a storage layer, and an application layer. The data source layer may include multiple data sources for providing raw data; the input layer is used to input raw data; the data processing layer is used to perform data cleaning, data deduplication, machine learning processing, etc. on the raw data to obtain an object label data table; the data storage layer may store data such as an object label data table, an object collection bitmap, and a label meta-information table; and the application layer may provide processing logic such as collection selection, retry, and alarm.

[0138] It can be seen that the data processing method provided in the embodiment of the present application adds tag data corresponding to the same tag type to the same object tag data table, and / or adds tag data corresponding to the same operation frequency range to the same object tag data table, thereby improving the scalability of the data table, and realizing on-demand storage, and improving data query efficiency; in addition, the object tag data table in the embodiment of the present application is indexed and sorted according to the tag priority of the object tags contained in the object tag data table, which is conducive to improving the execution efficiency of single-tag conditional statements, thereby improving the overall processing efficiency.

[0139] Embodiment 3

[0140] Figure 6 FIG. 1 shows a schematic diagram of the structure of a data processing device provided in Embodiment 3 of the present application. Figure 6 As shown, the device includes: a receiving module 610, a generating module 620 and an executing module 630.

[0141] The receiving module 610 is used to receive a creation request for a target object set sent by a client;

[0142] A generating module 620, configured to generate a single-label conditional statement and a multi-label aggregation statement corresponding to the creation request;

[0143] The execution module 630 is used to execute the single-label conditional statement to generate a single-label result bitmap; execute the multi-label aggregation statement to perform a bitmap operation on the single-label result bitmap to obtain an intermediate result bitmap, and perform a bitmap operation on the intermediate result bitmap and / or the single-label result bitmap to generate a target object set bitmap.

[0144] In an optional implementation, the generation module 620 is used to: parse the creation request, determine the target tags associated with the target object set, the constraints of the target tags, and the logical relationship between the target tags;

[0145] Querying the tag meta information matching the target tag and the constraint condition from a pre-stored tag meta information table;

[0146] For each target tag, a single-tag conditional statement of the target tag is generated according to the constraint conditions of the target tag, tag meta information and bitmap generation function;

[0147] Determine a bitmap operation function that matches the logical relationship, and generate the multi-label aggregation statement according to the bitmap operation function.

[0148] In an optional embodiment, the target tag includes: an object tag and / or a collection tag;

[0149] The generating module 620 is used to: generate, for an object tag, an object tag conditional statement corresponding to the object tag; and / or, for a set tag, generate a set tag conditional statement corresponding to the object tag;

[0150] The execution module 630 is used to: execute the object label conditional statement to generate a single label result bitmap based on a pre-stored object label data table; and / or execute the set label conditional statement to generate a single label result bitmap based on a pre-stored object set bitmap.

[0151] In an optional embodiment, the device also includes: a label data storage module (not shown in the figure), which is used to obtain initial label data; add label data corresponding to the same label type to the same object label data table; and / or add label data corresponding to the same operation frequency range to the same object label data table; wherein the number of labels contained in the object label data table is less than a preset number threshold.

[0152] In an optional implementation, the label storage module is used to: generate a data table index of the object label data table; wherein the data table index is sorted in a preset order.

[0153] In an optional implementation, the preset order is: the tag priority of the object tags included in the object tag data table.

[0154] In an optional implementation, the tag priority is obtained in the following manner: counting the operation frequencies of different object tags; and determining the tag priority of the object tag according to the operation frequencies.

[0155] It can be seen that the data processing device provided by the embodiment of the present application converts the creation request for the target object set sent by the client into a single-label conditional statement and a multi-label aggregation statement corresponding to each label. The single-label conditional statement can convert the processing result for a single label into bitmap storage, and the multi-label aggregation statement performs a bitmap operation based on the single-label result bitmap of the single-label conditional statement to obtain an intermediate result bitmap, and then obtains the target object set bitmap. Therefore, storage resources can be greatly saved through bitmap storage, and the integration of single-label conditional statements and multi-label aggregation statements with bitmap operations can greatly improve the efficiency of object set creation and improve data processing speed.

[0156] Embodiment 4

[0157] Figure 7 A schematic diagram of the structure of a computing device provided in Embodiment 4 of the present application is shown. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0158] like Figure 7 As shown, the computing device may include: a processor (processor) 702 , a communications interface (Communications Interface) 704 , a memory (memory) 706 , and a communication bus 708 .

[0159] The processor 702, the communication interface 704, and the memory 706 communicate with each other via a communication bus 708. The communication interface 704 is used to communicate with other devices such as a client or other server network elements. The processor 702 is used to execute a program 710, which can specifically execute the relevant steps in the above-mentioned data processing method embodiment for a computing device.

[0160] Specifically, the program 710 may include program codes, which include computer operation instructions.

[0161] The processor 702 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0162] The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The program 710 may be specifically used to enable the processor 702 to perform the operations in the above method embodiment.

[0163] Embodiment 5

[0164] Embodiment 5 of the present application provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction or computer program, and the executable instruction or computer program enables a processor to execute operations corresponding to the data processing method in any of the above method embodiments.

[0165] Embodiment 6

[0166] Embodiment 6 of the present application provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to execute operations corresponding to the data processing method in any of the above method embodiments.

[0167] In summary, according to the computing device, computer storage medium and computer program product provided by this embodiment, the creation request for the target object set sent by the client is converted into a single-label conditional statement and a multi-label aggregation statement corresponding to each label. The single-label conditional statement can convert the processing result for a single label into bitmap storage, and the multi-label aggregation statement performs a bitmap operation based on the single-label result bitmap of the single-label conditional statement to obtain an intermediate result bitmap, and then obtains the target object set bitmap. Therefore, storage resources can be greatly saved through bitmap storage, and the integration of single-label conditional statements and multi-label aggregation statements with bitmap operations can greatly improve the efficiency of object set creation and improve data processing speed.

[0168] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0169] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0170] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0171] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0172] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0173] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0174] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data processing method, characterized in that: include: Receive a creation request for a target object collection sent by a client; Generate a single-label conditional statement and a multi-label aggregation statement corresponding to the creation request; Executing the single-label conditional statement to generate a single-label result bitmap; The multi-label aggregation statement is executed to perform a bitmap operation on the single-label result bitmap to obtain an intermediate result bitmap, and a bitmap operation is performed on the intermediate result bitmap and / or the single-label result bitmap to generate a target object set bitmap.

2. The method according to claim 1, characterized in that The generating of the single-label conditional statement and the multi-label aggregation statement corresponding to the creation request includes: Parsing the creation request to determine target tags associated with the target object set, constraints of the target tags, and logical relationships between the target tags; Querying the tag meta information matching the target tag and the constraint condition from a pre-stored tag meta information table; For each target tag, a single-tag conditional statement of the target tag is generated according to the constraint conditions of the target tag, tag meta information and bitmap generation function; Determine a bitmap operation function that matches the logical relationship, and generate the multi-label aggregation statement according to the bitmap operation function.

3. The method according to claim 2, characterized in that The target tag includes: an object tag and / or a collection tag; Then the generating of the single label conditional statement corresponding to the creation request includes: for the object label, generating the object label conditional statement corresponding to the object label; and / or, for the set label, generating the set label conditional statement corresponding to the object label; Then executing the single-label conditional statement to generate a single-label result bitmap includes: based on a pre-stored object label data table, executing the object label conditional statement to generate a single-label result bitmap; and / or, based on a pre-stored object collection bitmap, executing the collection label conditional statement to generate a single-label result bitmap.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Get initial label data; Adding tag data corresponding to the same tag type to the same object tag data table; and / or, adding tag data corresponding to the same operation frequency range to the same object tag data table; The number of labels contained in the object label data table is less than a preset number threshold.

5. The method according to claim 4, characterized in that The method further comprises: generating a data table index of the object tag data table; Wherein, the data table indexes are sorted according to a preset order.

6. The method according to claim 5, characterized in that The preset order is: the tag priority of the object tags included in the object tag data table.

7. The method according to claim 6, characterized in that The method further comprises: Count the operation frequencies of different object tags; The tag priority of the object tag is determined according to the operation frequency.

8. A data processing device, characterized in that: include: A receiving module, used for receiving a creation request for a target object set sent by a client; A generation module, used to generate a single-label conditional statement and a multi-label aggregation statement corresponding to the creation request; An execution module, used for executing the single-label conditional statement to generate a single-label result bitmap; The multi-label aggregation statement is executed to perform a bitmap operation on the single-label result bitmap to obtain an intermediate result bitmap, and a bitmap operation is performed on the intermediate result bitmap and / or the single-label result bitmap to generate a target object set bitmap.

9. A computing device, characterized in that include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data processing method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the data processing method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises at least one executable instruction, wherein the executable instruction enables a processor to execute an operation corresponding to the data processing method according to any one of claims 1 to 7.

Images