A data processing method, system, storage medium and electronic device

Abstract

The application discloses a data processing method and system, a storage medium and an electronic device, converts obtained original data to obtain built-in label data, the original data is composed of tables in a data warehouse, the built-in label data is converted from the original data to be composed of tables containing user attributes, and a calculation framework of a MapReduce model is used to perform label bitmap construction and calculation on the built-in label data and external label data to obtain a people bitmap. Through the above scheme, after reading external label data, that is, external heterogeneous data sources, the calculation framework of the extended MapReduce model is used to support bitmap construction and calculation, and the calculation framework of the extended MapReduce model is integrated with an SQL language, so that the SQL language can be called in the process of bitmap construction and calculation, and the external heterogeneous data sources are involved in the process of people bitmap construction, thereby reducing the limitation of the people bitmap generation mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bitmap processing, and more particularly to a data processing method, system, storage medium and electronic device. BACKGROUND

[0002] Construction of a people bitmap is usually used in a user portrait system. In the user portrait system, a user generates a filtering condition through combination of different tags (such as gender, nationality, etc.), and obtains a user set meeting the condition in a database according to the filtering condition.

[0003] At present, tag generation of the user portrait system is mainly completed by a big data platform, and people generation is completed by an external database supporting bitmap calculation. Since the tag system has a large number of self-defined tags of business parties (users), the self-defined tags, unlike the tags in the system, exist only in the big data platform, and the self-defined tags are scattered in the storage of the business parties according to different technologies used by the business parties, such as MongoDB, HBase, MySQL, etc. databases, or big data platforms of the business parties. However, the above databases are difficult to directly interact with the external heterogeneous databases for bitmap calculation.

[0004] The self-defined tags are difficult to be integrated into the above data flow to participate in people bitmap calculation, so that the self-defined tags are difficult to be combined and filtered with the built-in tags of the user portrait system, and can only generate a people bitmap through the built-in tags, thereby limiting the way of generating the people bitmap.

[0005] Therefore, the existing way of generating a people bitmap has high limitations. SUMMARY

[0006] Therefore, the present application discloses a data processing method, system, storage medium and electronic device, which realizes the purpose of reducing the limitations of the way of generating a people bitmap.

[0007] In order to achieve the above purpose, the disclosed technical solution is as follows:

[0008] The first aspect of the present application discloses a data processing method, which comprises:

[0009] Converting the obtained original data to obtain built-in tag data, wherein the original data is composed of tables in a data warehouse, and the built-in tag data is composed of tables containing user attributes converted from the original data;

[0010] A MapReduce-like computing framework is used to perform tag bitmap processing on the built-in tag data and the acquired user-defined external tag data to obtain a crowd bitmap. Preferably, the above operations are performed using a preset SQL language and a preset MapReduce-like computing framework.

[0011] Preferably, the step of converting the acquired raw data to obtain built-in tag data includes:

[0012] The acquired raw data is transformed using a preset conversion script to obtain built-in tag data.

[0013] Preferably, the process of obtaining user-defined external tag data includes:

[0014] The system obtains user-defined external tag data through a preset calculation framework.

[0015] Preferably, a MapReduce-like computing framework is used to perform tag bitmap processing on the built-in tag data and the acquired user-defined external tag data to obtain a crowd bitmap, including:

[0016] Create first-class and second-class extensions for the computational framework of the MapReduce model;

[0017] Using a preset SQL language and the first type of extension, a bitmap is constructed from the built-in tag data and the acquired user-defined external tag data to obtain the bitmap to be processed;

[0018] Using a preset SQL language and the second type of extension, bitwise operations are performed on the bitmap to be processed to obtain a crowd bitmap.

[0019] A second aspect of the present invention discloses a data processing system, the system comprising:

[0020] The conversion unit is used to convert the acquired raw data to obtain built-in tag data. The raw data consists of tables in the data warehouse, and the built-in tag data is converted from the raw data into tables containing user attributes.

[0021] The processing unit is used to perform tag bitmap processing operations on the built-in tag data and the acquired user-defined external tag data using a MapReduce-like computing framework to obtain a crowd bitmap.

[0022] Preferably, the conversion unit is specifically used for:

[0023] The acquired raw data is transformed using a preset conversion script to obtain built-in tag data.

[0024] Preferably, the processing unit in the process of acquiring user-defined external tag data is specifically used for:

[0025] The system obtains user-defined external tag data through a preset calculation framework.

[0026] Preferably, the processing unit includes:

[0027] Create modules to create first-class and second-class extensions to the computational framework of the MapReduce model;

[0028] The construction module is used to construct a bitmap from the built-in tag data and the acquired user-defined external tag data using a preset SQL language and the first type of extension, so as to obtain a bitmap to be processed.

[0029] The calculation module is used to perform bit operations on the bitmap to be processed using a preset SQL language and the second type of extension to obtain a crowd bitmap.

[0030] A third aspect of the present invention discloses a storage medium comprising stored instructions, wherein, when the instructions are executed, the device in which the storage medium is located is controlled to perform a data processing method as described in any one of the first aspects.

[0031] The fourth aspect of the present invention discloses an electronic device, including a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors using the data processing method as described in any one of the first aspects.

[0032] As described in the above technical solution, the acquired raw data is transformed to obtain built-in tags. The raw data consists of tables in the data warehouse, while the built-in tags consist of tables containing user attributes in the data mart. User-defined external tags are obtained. A MapReduce-like computing framework is used to construct and calculate a tag bitmap from the built-in and external tags, resulting in a crowd bitmap. Through this solution, after reading the external tags (external heterogeneous data source) based on the preset computing framework, the external tag data (i.e., the external heterogeneous data source) is read. The MapReduce-like computing framework is extended to support bitmap construction and calculation. Furthermore, it is integrated with the extended MapReduce-like computing framework using SQL, allowing the bitmap construction and calculation process to be called via SQL. This enables external heterogeneous data sources to participate in the crowd bitmap construction process, reducing the limitations of the current crowd bitmap generation method. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating a data processing method disclosed in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of the MapReduce computing model disclosed in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the process for obtaining a crowd bitmap as disclosed in an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of a data processing system disclosed in an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0041] As can be seen from the background technology, existing methods for generating crowd bitmaps have significant limitations.

[0042] To address this issue, embodiments of the present invention disclose a data processing method, system, storage medium, and electronic device. After reading external tag data (i.e., external heterogeneous data sources), the method extends a MapReduce-like computational framework to support bitmap construction and computation. It integrates the extended MapReduce-like computational framework with SQL, enabling SQL calls during bitmap construction and computation. This allows external heterogeneous data sources to participate in the crowd bitmap construction process, reducing the limitations of previous methods for generating crowd bitmaps. The specific implementation is described in detail in the following embodiments.

[0043] All user information involved in the crowd bitmap processing in this solution has been authorized by the users in advance, and the data processing complies with relevant laws and regulations.

[0044] refer to Figure 1 The diagram shown is a flowchart of a data processing method disclosed in an embodiment of the present invention. The data processing method mainly includes the following steps:

[0045] Step S101: Transform the obtained raw data to obtain built-in tag data. The raw data consists of tables in the data warehouse, and the built-in tag data is formed by transforming the raw data into tables containing user attributes.

[0046] In step S101, the acquired raw data is converted using a preset conversion script to obtain built-in tag data, which is the system tag data of the user profiling system.

[0047] Built-in tag data is stored in a data mart.

[0048] The preset conversion script can be a JavaScript Object Notation (JSON) script or other conversion scripts. The preset conversion script shall be selected by the technical personnel according to the actual situation, and the present invention does not impose any specific limitations.

[0049] The process of building built-in tags all takes place within big data platforms, most of which are built on Hadoop. Hadoop also supports various computing frameworks, such as Flink, MapReduce, and Spark. These computing frameworks mostly support direct reading from commercially available data stores. Furthermore, if a direct reading solution is not available, any data source can be adapted through a custom source. Building upon this foundation, enabling these big data frameworks to support bitmap computation allows external heterogeneous data sources to participate in the process of building crowd bitmaps, reducing the limitations of generating crowd bitmaps.

[0050] User attributes can include user gender, user age, and so on.

[0051] Currently, computing frameworks such as Flink and Spark support direct access to data sources including HBase, Elasticsearch, MongoDB, MySQL, Redis, PostgreSQL, ClickHouse, Cassandra, TiDB, and other databases, covering NoSQL, NewSQL, and traditional relational databases.

[0052] A data warehouse is an integrated, subject-oriented collection of data designed to support the functionality of a Decision Support System (DSS). In a data warehouse, each data unit is associated with a specific time period. A data warehouse includes atomic-level data and lightly summarized data. It is a subject-oriented, integrated, non-updatable (stable), and time-varying (different times) collection of data.

[0053] The structure of data in a data mart is often described as a star schema or snowflake schema. A star schema consists of two basic parts: a fact table and various supporting dimension tables.

[0054] Step S102: Obtain user-defined external tag data.

[0055] In step S102, user-defined external tag data is obtained through a preset calculation framework.

[0056] External tag data is typically stored in other storage specified by the user, such as relational or non-relational databases.

[0057] The preset computing framework can be Flink, MapReduce, Spark, etc. The specific preset computing framework is determined by the technical personnel according to the actual situation, and this invention does not impose any specific limitations.

[0058] External tag data is used to identify external heterogeneous data sources.

[0059] To make it easier to understand external tag data, an example is provided below:

[0060] For example, the external tag data code is as follows:

[0061] selectbuildbitmap(user_id)rbm

[0062] fromEXTERNAL_user_basic_info where age="20-30"

[0063] The tag 'age='20-30' belongs to an external heterogeneous data source, i.e., external tag data.

[0064] Steps S101 and S102 can be processed in parallel.

[0065] Step S103: Using a MapReduce-like computing framework, perform tag bitmap processing on the built-in tag data and the acquired user-defined external tag data to obtain a crowd bitmap.

[0066] In step S103, the extended MapReduce-like computing framework is used to construct a bitmap and perform bitmap operations on the built-in tag data and the acquired user-defined external tag data to obtain a crowd bitmap. Preferably, the above operations are performed using a preset SQL language and a MapReduce-like computing framework.

[0067] The MapReduce-like computing framework is a high-performance, cluster-based parallel computing platform (Cluster Infrastructure). It allows the construction of distributed and parallel computing clusters containing tens, hundreds, or even thousands of nodes using ordinary commercial servers. It is also a parallel computing and execution software framework. It provides a large but well-designed parallel computing software framework that automatically performs parallel processing of computational tasks, automatically partitions computational data and tasks, automatically allocates and executes tasks on cluster nodes, and collects computational results. It delegates many complex low-level system details involved in parallel computing, such as data distribution and storage, data communication, and fault tolerance, to the system, greatly reducing the burden on software developers. Furthermore, it is a parallel programming model and methodology. Leveraging the design principles of the functional programming language Lisp, it provides a simple method for parallel programming, using the Map and Reduce functions to implement basic parallel computing tasks. It provides abstract operations and parallel programming interfaces for easy and convenient programming and computation of large-scale data.

[0068] The extended MapReduce-like computing framework enables bitmap construction and computation capabilities. It can directly obtain data from local data warehouses and external data sources for bitmap computation. The data in the data warehouse and external data sources are labeled datasets, which are collections (tables) of mappings of label values.

[0069] Extensions to the MapReduce-like computing framework include first-class extensions and second-class extensions.

[0070] The first type of extension is used to construct a bitmap from the built-in tag data and the acquired user-defined external tag data to obtain the bitmap to be processed.

[0071] The second type of extension is used to perform bit operations on the bitmap to be processed to obtain a crowd bitmap. The second type of extension has four types of calculations: and, or, xor, and not.

[0072] Specifically, using a MapReduce-like computing framework, the process of constructing a bitmap and performing bitmap operations on the built-in tag data and the acquired user-defined external tag data to obtain the crowd bitmap is as follows:

[0073] First, a first-class extension and a second-class extension are created for the MapReduce-like computing framework. Then, using the preset SQL language and the first-class extension, a bitmap is constructed from the built-in tag data and the acquired user-defined external tag data to obtain the bitmap to be processed. Finally, using the preset SQL language and the second-class extension, bit operations are performed on the bitmap to be processed to obtain the crowd bitmap.

[0074] To better understand the advantages of crowd-based bitmaps, here's an example:

[0075] Assuming there are 1 billion users, each with a unique ID of type int (4 bytes, 32 bits), storing these user IDs alone would require 3.72 GB of memory (4 * 1000000000 / 1024 / 1024 / 1024). For 4 billion users, this would require 16 GB of memory. This would place a significant computational burden on the application when performing tag combination filtering. Therefore, in scenarios involving multi-tag combination filtering, a Bitmap data structure is typically chosen to store tags, and the user group is generated using the bitmap. In a program, the smallest unit we can access is a bit, which can only store 0 or 1. Using bits to represent whether a user matches a tag, a user group of 1 billion would only require about 120 MB, a 32-fold difference in size. Because bitmap-based crowd generation has a small storage capacity, and because bitmap calculations directly use bitwise operations, which are very low-level computer operations, unnecessary computational costs can be reduced. In summary, using bitmaps to represent crowds can reduce computational pressure from both input and computational costs, significantly improving computational efficiency. Compared with conventional calculation methods, computational efficiency is greatly improved.

[0076] refer to Figure 2 The diagram shown is a structural schematic of a computational framework similar to the MapReduce model.

[0077] Map Tasks can be viewed as single-row processing, while Reduce Tasks can be considered as aggregation processing of the results from Map Tasks. MapReduce-like computing frameworks typically provide interfaces similar to Map and Reduce for developers to implement their own logic.

[0078] First type of extension:

[0079] In MapTask, for a single line, the input should be an integer, and the result should be a byte array (bitmap) to store the bit information represented by that integer. The pseudocode is as follows:

[0080] Func map(x:int):

[0081] ReturnArrayBytes(x)

[0082] In ReduceTask, multiple byte arrays need to be converted into bitmap data structures first, and then merged using the bitmap data structure merging function. The final result is a single bitmap. If it is necessary to save the constructed bitmap to be processed, the bitmap to be processed needs to be serialized to the file system. The pseudocode is as follows:

[0083] Func reduce(x:ArrayBytes,y:ArrayBytes):

[0084] Bm_x = Bitmap(x)

[0085] Bm_y = Bitmap(y)

[0086] Return Bm_x.merge(Bm_y)

[0087] Second type of extension:

[0088] When performing bitmap calculations, the bitmap to be calculated needs to be inserted into a column, that is, one bitmap per row, so no additional operations are needed in the map phase.

[0089] Funcmap(x:Bitmap):

[0090] ReturnBitmap(x)

[0091] During the reduce phase, bitmap computation is performed.

[0092] Funcreduce(x:Bitmap, y:Bitmap):

[0093] Return x.compute(y)

[0094] Here, compute represents performing bitwise operations between one bitmap and another bitmap, and there are four types: and, or, xor, and not.

[0095] After implementing the above interfaces for different computing engines, it is necessary to integrate them with their respective SQL methods. This will allow the computation process to be invoked via SQL. There are two integration methods:

[0096] Integration with the computing framework requires modifying the computing engine's source code files to add these two types of functions and register them through a function registry. This approach allows users to use the functions directly as native functions without having to register them each time they are used.

[0097] Users need to define and register both the first and second type of extensions when using them.

[0098] There are two ways to register: one is to register once at the beginning of each script, and then no further registration is needed within that script; the other is to register once directly in the engine, which is equivalent to modifying the engine source code, and no further registration is needed for any subsequent use.

[0099] After registration, you can construct and calculate bitmaps using the preset SQL language, which is the process of generating the user population. The code is as follows:

[0100] select bm_andnot(t2.rbm)from(

[0101] select bm_or(t1.rbm)rbm from(

[0102] select bm_and(t0.rbm)rbm from(

[0103] selectbuildbitmap(user_id)rbm

[0104] fromuser_basic_info where sex="male"

[0105] union all

[0106] selectbuildbitmap(user_id)rbm

[0107] fromEXTERNAL_user_basic_info where age="20-30"

[0108] )t0

[0109] union all

[0110] selectbuildbitmap(user_id)rbm

[0111] fromuser_basic_info whereborn_address="Beijing"

[0112] )t1

[0113] union all

[0114] selectbuildbitmap(user_id)rbm

[0115] fromuser_basic_info wheredgree="Master"

[0116] )t2

[0117] This SQL statement represents individuals who are male and (and) aged 20-30 or (or) born in Beijing, excluding those with a master's degree. Here, `buildbitmap` refers to the bitmap construction function, and `bm_andnot`, `bm_and`, and `bm_or` represent the three types of the `Compute` function (there is also `bm_xor`). The SQL statement shows that the tag `age='20-30'` belongs to an external heterogeneous data source (`EXTERNAL_user_basic_info`), meaning it's an external tag. Internal tags can be `sex`, `address`, etc.

[0118] For data sources that are not directly supported, the data source module of the computing framework of this type of MapReduce model can be extended in a custom way to achieve the purpose of adapting to the data source.

[0119] To better understand the process of generating a crowd bitmap, an example is provided below:

[0120] For example, common characteristics of a group of people, such as being over 30 years old, having a master's degree, and having placed their first order within the last seven days, are considered external tags. These external tags can be combined using operators such as OR, AND, and NOT to generate a user demographic. For instance, if someone is over 30 years old and has a master's degree, removing the "last seven days" requirement for their first order, we obtain a user demographic bitmap.

[0121] In this embodiment of the invention, after reading external tag data, i.e., external heterogeneous data sources, based on a preset computing framework, the computing framework of the MapReduce-like model is extended in a custom way to support bitmap construction and computation. The extended MapReduce-like model computing framework is then integrated with SQL language, so that the bitmap construction and computation process can be called through SQL language. This allows external heterogeneous data sources to participate in the process of building the crowd bitmap, reducing the limitations of the crowd bitmap generation method.

[0122] refer to Figure 3 As shown, step S103 above involves a computing framework using a MapReduce-like model to perform bitmap processing operations on the built-in tag data and the acquired user-defined external tag data to obtain a crowd bitmap. This mainly includes the following steps:

[0123] Step S301: Create a first-class extension and a second-class extension for the computational framework of the MapReduce model.

[0124] Step S302: Using the preset SQL language and the first type of extension, construct a bitmap from the built-in tag data and the obtained user-defined external tag data to obtain the bitmap to be processed.

[0125] Step S303: Perform bit operations on the bitmap to be processed using the preset SQL language and the second type of extension to obtain the crowd bitmap.

[0126] The execution principle of steps S301-S303 is the same as that of step S103 above, and can be referred to. It will not be repeated here.

[0127] In this embodiment of the invention, a bitmap is constructed from built-in tag data and external tag data using a preset SQL language and a first type of extension to obtain a bitmap to be processed. Then, bitmap calculations are performed on the bitmap to be processed using a preset SQL language and a second type of extension to obtain a crowd bitmap. This allows the bitmap construction and calculation process to be called using a preset SQL language, enabling external heterogeneous data sources to participate in the crowd bitmap construction process and reducing the limitations of the crowd bitmap generation method.

[0128] Based on the above embodiments Figure 1 The present invention discloses a data processing method, and also provides a corresponding structural schematic diagram of a data processing system, as shown in the embodiments below. Figure 4 As shown, the data processing system includes a conversion unit 401 and a processing unit 402.

[0129] The conversion unit 401 is used to convert the acquired raw data to obtain built-in tag data. The raw data consists of tables in the data warehouse, and the built-in tag data is converted from the raw data into tables containing user attributes.

[0130] The system uses a pre-defined conversion script to convert the acquired raw data into built-in tag data, which serves as the system tags for the user profiling system.

[0131] Processing unit 402 is used to perform tag bitmap processing operations on built-in tag data and acquired user-defined external tag data using a computing framework similar to MapReduce, to obtain a crowd bitmap.

[0132] Specifically, using an extended MapReduce-like computational framework, bitmap construction and bitmap operations are performed on the built-in tag data and the acquired user-defined external tag data to obtain a crowd bitmap. Preferably, the above operations are performed using a preset SQL language and a MapReduce-like computational framework.

[0133] Furthermore, the conversion unit 401 is specifically used to convert the acquired raw data using a preset conversion script to obtain built-in tag data.

[0134] Furthermore, the processing unit 402 for obtaining user-defined external tag data is specifically used to obtain user-defined external tag data through a preset calculation framework.

[0135] Furthermore, the processing unit 402 includes a building module and a computing module.

[0136] Create modules to create first-class and second-class extensions to the computational framework of the MapReduce model.

[0137] The construction module is used to construct bitmaps from built-in tag data and acquired user-defined external tag data using a preset SQL language and the first type of extension, so as to obtain the bitmap to be processed.

[0138] The calculation module is used to perform bitmap calculations on the bitmap to be processed using a preset SQL language and a second type of extension, to obtain a crowd bitmap.

[0139] Among them, there are four types of calculation types for bit operations on the bitmap to be processed: and, or, xor, and andNot.

[0140] In this embodiment of the invention, after reading external tag data, i.e., external heterogeneous data sources, based on a preset computing framework, the computing framework of the MapReduce-like model is extended in a customized way to support bitmap construction and calculation. The extended MapReduce-like model computing framework is integrated with a preset SQL language, so that the bitmap construction and calculation process can be called through the preset SQL language. This allows external heterogeneous data sources to participate in the process of building the crowd bitmap, reducing the limitations of the crowd bitmap generation method.

[0141] This invention also provides a storage medium, which includes stored instructions, wherein the instructions, when executed, control the device where the storage medium is located to perform the above-described data processing method.

[0142] This invention also provides an electronic device, the structural schematic of which is shown below. Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein one or more instructions 502 are stored in the memory 501 and configured to be executed by one or more processors 503 to perform the following operations:

[0143] The acquired raw data is transformed to obtain built-in tag data. The raw data consists of tables in the data warehouse, and the built-in tag data is formed by transforming the raw data into tables containing user attributes.

[0144] A MapReduce-like computing framework is used to perform bitmap processing on built-in tag data and acquired user-defined external tag data to obtain a crowd bitmap.

[0145] The specific implementation processes and derivative methods of the above embodiments are all within the protection scope of this invention.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0147] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0148] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: The acquired raw data is transformed to obtain built-in tag data. The raw data consists of tables in the data warehouse, and the built-in tag data is formed by transforming the raw data into tables containing user attributes. A MapReduce-like computing framework is used to perform tag bitmap processing operations on the built-in tag data and the acquired user-defined external tag data to obtain a crowd bitmap. This includes: creating a first type of extension and a second type of extension on the MapReduce-like computing framework; constructing a bitmap from the built-in tag data and the acquired user-defined external tag data using a preset SQL language and the first type of extension to obtain a bitmap to be processed; and performing bitwise operations on the bitmap to be processed using a preset SQL language and the second type of extension to obtain the crowd bitmap. The second type of extension has four calculation types: AND, OR, XOR, and NOT. The computational framework of the MapReduce-like model is a high-performance parallel computing platform based on a cluster, belonging to the category of parallel computing and runtime software frameworks; the cluster is a distributed and parallel computing cluster containing multiple nodes; the computational framework of the MapReduce-like model includes multiple parallel Map tasks and Reduce tasks. The step of constructing a bitmap from the built-in tag data and the acquired user-defined external tag data using a preset SQL language and the first type of extension, to obtain the bitmap to be processed, includes: Using a preset SQL language and the first type of extension, the built-in tag data and the acquired user-defined external tag data are processed row by row to convert the built-in tag data and the acquired user-defined external tag data into corresponding byte arrays; multiple byte arrays are merged to obtain the bitmap to be processed; The step of performing bitwise operations on the bitmap to be processed using a preset SQL language and the second type of extension to obtain a crowd bitmap includes: Using a preset SQL language and the second type of extension, the bitmap to be processed is inserted into the corresponding column; bit operations are performed on the bitmap to be processed to obtain the crowd bitmap.

2. The method according to claim 1, characterized in that, The process of transforming the acquired raw data to obtain built-in tag data includes: The acquired raw data is transformed using a preset conversion script to obtain built-in tag data.

3. The method according to claim 1, characterized in that, The process of obtaining user-defined external tag data includes: The system obtains user-defined external tag data through a preset calculation framework.

4. A data processing system, characterized in that, The system includes: The conversion unit is used to convert the acquired raw data to obtain built-in tag data. The raw data consists of tables in the data warehouse, and the built-in tag data is converted from the raw data into tables containing user attributes. The processing unit uses a MapReduce-like computing framework to perform tag bitmap processing on the built-in tag data and the acquired user-defined external tag data to obtain a crowd bitmap. The processing unit includes: a creation module, a construction module, and a calculation module; The creation module is used to create a first type of extension and a second type of extension for the computing framework of the MapReduce model; The construction module is used to construct a bitmap from the built-in tag data and the acquired user-defined external tag data using a preset SQL language and the first type of extension, so as to obtain a bitmap to be processed. The calculation module is used to perform bit operations on the bitmap to be processed using a preset SQL language and the second type of extension to obtain a crowd bitmap; the second type of extension has four types of calculations: AND, OR, XOR, and NOT. The computational framework of the MapReduce-like model is a high-performance parallel computing platform based on a cluster, belonging to the category of parallel computing and runtime software frameworks; the cluster is a distributed and parallel computing cluster containing multiple nodes; the computational framework of the MapReduce-like model includes multiple parallel Map tasks and Reduce tasks. The step of constructing a bitmap from the built-in tag data and the acquired user-defined external tag data using a preset SQL language and the first type of extension, to obtain the bitmap to be processed, includes: Using a preset SQL language and the first type of extension, the built-in tag data and the acquired user-defined external tag data are processed row by row to convert the built-in tag data and the acquired user-defined external tag data into corresponding byte arrays; multiple byte arrays are merged to obtain the bitmap to be processed; The step of performing bitwise operations on the bitmap to be processed using a preset SQL language and the second type of extension to obtain a crowd bitmap includes: Using a preset SQL language and the second type of extension, the bitmap to be processed is inserted into the corresponding column; bit operations are performed on the bitmap to be processed to obtain the crowd bitmap.

5. The system according to claim 4, characterized in that, The conversion unit is specifically used for: The acquired raw data is transformed using a preset conversion script to obtain built-in tag data.

6. The system according to claim 4, characterized in that, The processing unit in the process of acquiring user-defined external tag data is specifically used for: The system obtains user-defined external tag data through a preset calculation framework.

7. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device in which the storage medium is located is controlled to perform the data processing method as described in any one of claims 1 to 3.

8. An electronic device, characterized in that, It includes a memory, and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors as described in any one of claims 1 to 3.

Images