Data processing method and system, electronic device and storage medium
By building a bitmap function and tag integration layer in Hive, and automatically generating query statements with front-end application plug-ins, the problems of low big data retrieval efficiency and easy data confusion are solved, fast query and efficient reuse are achieved, and data processing is improved. Efficiency and convenience.
Patent Information
- Application Number
- CN202410154113.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-02-02
AI Technical Summary
When processing big data in the prior art, bitmap indexing is difficult to improve the retrieval efficiency of deep business logic, and query engines are prone to data confusion when the amount of data is large, resulting in low query efficiency.
Through the extended support provided by Hive, we build bitmap functions and tag integration layers, combine the front-end application plug-in to generate data applications, and automatically generate query statements based on circle selection tags to achieve rapid data query and multiplexing.
It improves the efficiency and reuse rate of data query, reduces the difficulty of data understanding, expands the convenience and utilization of data, and reduces the error rate of manual writing of query statements.
Smart Images

Figure CN118069680B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet big data technology, and in particular to a data processing method and system, an electronic device and a storage medium. Background Art
[0002] With the development of Internet technology and the advent of the big data era, in some application scenarios, more and more data needs to be stored and processed, which puts higher requirements on data storage, retrieval, and processing. In related technologies, bitmap indexing is used to improve retrieval efficiency. However, this solution is oriented to single metadata and has poor retrieval efficiency for deep business logic. In another solution, data query is performed through a query engine. This solution cannot alleviate the problem of data confusion when the data volume is large, and the data query efficiency is not high. Summary of the invention
[0003] The main purpose of the embodiments of the present application is to provide an efficient data processing method and system, electronic device and storage medium.
[0004] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application proposes a data processing method, the method comprising: constructing a bitmap function according to the extended support provided by Hive; obtaining a data set related to business processing, and constructing a label integration layer according to the data set; the label integration layer is used to trace back label detail data; making interactive events for front-end application plug-ins, generating data applications based on the label integration layer; based on the data application, receiving object circled labels, determining data query statements, and determining business data according to the data query statements. The embodiment of the present application can realize automatic generation of query statements based on circled labels by combining the label integration layer with bitmap functions, realize rapid query of data, and improve query efficiency and data reuse rate.
[0005] In some embodiments, the method provided in the embodiments of the present application, constructing a bitmap function according to the extended support provided by Hive, includes:
[0006] Create new classes;
[0007] Using the first type of objects and the second type of objects as bitmap carriers, performing set operations on the first type of objects and the second type of objects, and determining a bitmap operation result set;
[0008] The new class is encapsulated and registered with the Hive to construct a bitmap function.
[0009] In some embodiments, the method provided by the embodiments of the present application, the step of obtaining a data set related to the business process and constructing a tag integration layer according to the data set includes:
[0010] Get the data set of the detail layer of the data warehouse;
[0011] Based on the data set, determine the subject details of the business requirements;
[0012] Establishing a global unified identifier for the subject details and constructing a tag detail layer;
[0013] According to the label detail layer and data application classification, the data is classified and clustered to determine the label integration layer.
[0014] In some embodiments, the method provided by the embodiments of the present application further includes:
[0015] Creating a control in the data application through the front-end application plug-in;
[0016] Through the front-end application plug-in, the selected nodes are converted into a data exchange structure in the data application; the data exchange structure includes the selected tag name, logical relationship and tag value; the tag name is used to represent the tag table of the tag integration layer, and the tag value is used to represent the dimension value of the tag table.
[0017] In some embodiments, the method provided by the embodiment of the present application, the interactive event for the front-end application plug-in, and the generation of the data application based on the tag integration layer, includes:
[0018] Expanding the tag type based on the tag integration layer to establish an interaction event;
[0019] Defining the logical relationship of the tags according to the relationship in the bitmap function;
[0020] Prompts and slogans are added to the interactive interface of the data application to improve the data application.
[0021] In some embodiments, the method provided by the embodiments of the present application further includes:
[0022] According to the preset cut-off value, the group is classified based on the consumption data of the object;
[0023] Create a global unified identifier for each object and convert the classification results into a bitmap to build a population model;
[0024] According to the bitmap function and the object selection label, the objects in the group model are grouped and cross-calculated to determine the index data of each group.
[0025] In some embodiments, the method provided by the embodiments of the present application further includes:
[0026] Parsing the circled nodes according to the tag name, the logical relationship and the tag value, and determining a bitmap data set through a cross operation of the bitmap function;
[0027] Determining a target global unified identifier according to the bitmap data set;
[0028] According to the target global unified identifier, the target subject details are determined based on the tag detail layer; the target subject details are subject details of the target object.
[0029] To achieve the above object, another aspect of an embodiment of the present application provides a data processing system, the system comprising:
[0030] The first module is used to build a bitmap function based on the extended support provided by Hive;
[0031] The second module is used to obtain a data set related to the business processing and construct a label integration layer according to the data set; the label integration layer is used to trace back the label detail data;
[0032] The third module is used to interact with the front-end application plug-in and generate data applications based on the tag integration layer;
[0033] The fourth module is used to receive the object circle selection label based on the data application, determine the data query statement, and determine the business data according to the data query statement.
[0034] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.
[0035] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0036] The embodiments of the present application include at least the following beneficial effects: The method provided in the embodiments of the present application includes: constructing a bitmap function according to the extended support provided by Hive; obtaining a data set related to the business processing, and constructing a label integration layer according to the data set; the label integration layer is used to trace back label detail data; making interactive events for the front-end application plug-in, generating a data application based on the label integration layer; based on the data application, receiving the object circled label, determining the data query statement, and determining the business data according to the data query statement. The embodiments of the present application can realize the automatic generation of query statements based on circled labels by combining the label integration layer with the bitmap function, realize rapid query of data, and improve query efficiency and data reuse rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1is a flow chart of an embodiment of the data processing method provided by the present application;
[0038] Figure 2 is a flow chart of another embodiment of the data processing method provided by the present application;
[0039] Figure 3 It is a flowchart of an embodiment of the bitmap function operation process provided by the present application;
[0040] Figure 4 is a flowchart of an embodiment of a process for establishing a label integration layer provided by the present application;
[0041] Figure 5 is a structural diagram of an embodiment of a tag information table provided by the present application;
[0042] Figure 6 This is a flowchart of an embodiment of label parsing provided by the present application;
[0043] Figure 7 is a flow chart of an embodiment of a crowd distribution process provided by the present application;
[0044] Figure 8 is a flow chart of another embodiment of the data encryption process provided by the present application;
[0045] Fig. 9 is a structural diagram of a data processing system provided in an embodiment of the present application;
[0046] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.
[0048] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".
[0049] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.
[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0051] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.
[0052] Trino (formerly known as Presto): is a fast distributed SQL query engine for big data analysis. It is designed to efficiently process large amounts of data and supports multiple data sources, including traditional relational databases and other data sources. It can complete federated queries across different data sources; the application scenarios mainly involve data warehouses and analytical work, such as data analysis, aggregating large amounts of data and generating reports.
[0053] UDF: It is a concept in Hive, which stands for User-Defined Function. It can be used to perform various calculations, conversions, and processing operations. By writing custom functions, Hive query functions can be extended to improve query flexibility and scalability.
[0054] Bitmap: A data structure that stores data in bits, where each bit represents an element. By using binary bits, it can compress an integer into one eighth of a byte, thus greatly saving memory space. This makes it highly efficient and space-saving when storing and processing large amounts of data.
[0055] RFM population: is a market segmentation concept based on three key indicators: the last purchase date (recency), purchase frequency (frequency) and purchase amount (monetary). RFM population analysis can help companies identify which customers are high-value customers and which are low-value customers, so as to develop targeted marketing strategies.
[0056] In the related technology, a solution uses a full-text search method with a bitmap index to achieve a more efficient full-text index. At the same time, due to the use of a bitmap index, the performance of the full-text search is also significantly improved to a great extent. This solution uses a bitmap to improve retrieval efficiency. It is mainly based on a single metadata, which is relatively one-sided, not suitable for retrieval of deep business logic, and is not very convenient. Another solution is to parse SQL query information based on the Presto query engine, and automatically switch to querying Hive through Presto to achieve downgrade. This solution only stores specific data in different directions. When the amount of stored data increases, model confusion is still prone to occur, which is not conducive to data query.
[0057] In this regard, the embodiment of the present application builds an integration layer based on a bitmap, and realizes interactive business query based on the collected model information combined with the bitmap function, which improves the retrieval efficiency while increasing the reuse rate of the model and reducing the application cost of big data. The embodiment of the present application extracts the label details from the detailed layer model data and then integrates the labels, so that the model meaning represented by the labels can be viewed intuitively, and Trino query statements can be automatically established through circle selection rules, which reduces the difficulty of understanding the data, expands the group that can be used, and greatly improves the convenience and utilization of data.
[0058] The data processing method provided in the embodiment of the present application relates to the field of Internet big data technology. The data processing method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements the data processing method, etc., but is not limited to the above forms.
[0059] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0060] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0061] Figure 1 is an optional flow chart of the data processing method provided in the embodiment of the present application; Figure 1 The method may include but is not limited to steps S100 to S400.
[0062] Step S100, constructing a bitmap function according to the extended support provided by Hive;
[0063] Step S200, obtaining a data set related to business processing, and constructing a label integration layer according to the data set; the label integration layer is used to trace back label detail data;
[0064] Step S300, making interactive events for the front-end application plug-in and generating data applications based on the tag integration layer;
[0065] Step S400, based on the data application, receiving the object selection tag, determining the data query statement, and determining the business data according to the data query statement.
[0066] Reference Figure 2As shown, the overall implementation process of the embodiment of the present application includes three parts, namely, bitmap function construction, data model establishment and data application. In the embodiment of the present application, the data set related to the business can be determined according to the actual application scenario. The embodiment of the present application can be used to determine the distribution of the population, and can also be used for recommendations in various scenarios. The present application does not limit the specific application scenarios of data processing. The embodiment of the present application reduces the threshold for the use of bitmaps and improves the efficiency of data application. By establishing a bitmap data model layer and bitmap calculation rules of subdivided dimensions, a statement for querying bitmap data can be automatically generated based on the selection conditions, thereby realizing rapid retrieval and rapid application of data. The embodiment of the present application increases the data reuse rate, and by extending the bitmap function that supports and, or, and and quantity statistics, combined with the constructed integration layer label model, the free combination of model data based on front-end interaction is realized, thereby improving the reuse rate. The embodiment of the present application reduces the error rate of manually written query statements. The front-end interaction is constructed by data label model information and definition rules, and query statements are generated by parsing the selection conditions, thereby reducing the number of manual writing and improving the accuracy of statements. The embodiments of the present application support non-professionals to use big data. By providing a simple and clear interface interaction on the front end, business personnel can freely combine model data based on model label options to obtain query results under the selected conditions.
[0067] In some embodiments, the method provided in the embodiments of the present application, based on the extended support provided by Hive, constructs a bitmap function, including:
[0068] Create new classes;
[0069] Using the first-category object and the second-category object as bitmap carriers, performing set operations on the first-category object and the second-category object, and determining a bitmap operation result set;
[0070] Encapsulate the new class, register it with Hive, and build the bitmap function.
[0071] In some possible implementations, the new class includes a first class object and a second class object. Specifically, the bitmap function construction of the present application can be implemented by using the GenericUDAFEvaluator extension support provided by Hive and using Java class inheritance to construct a UDF bitmap function. Figure 3As shown: Create a new class that inherits and extends, override the abstract method of the parent class, use two BitSet objects A and B as bitmap carriers, and then implement the intersection operation of the two BitSet objects to obtain the result set of the bitmap and operation, encapsulate the class and register it to Hive as a new function. Similarly, construct bitmap OR and NOT logical functions, as well as bitmap value conversion, bitmap aggregation statistics and other functions. Currently, Trino supports Hive's UDF function extension. The main step is to upload the encapsulated JAR package of the new Hive function as a plug-in to Trino to support Hive functions. Therefore, bitmap functions can be used in Trino to process Hive data.
[0072] In some embodiments, reference Figure 4 As shown, the method provided in the embodiment of the present application obtains a data set related to business processing, and constructs a label integration layer according to the data set, including:
[0073] Step S210, obtaining a data set at a detail layer of a data warehouse;
[0074] Step S220, determining the subject details of the business requirements according to the data set;
[0075] Step S230, establishing a global unified identifier for the subject details and constructing a tag detail layer;
[0076] Step S240, classify and aggregate the data according to the label detail layer and the data application classification, and determine the label integration layer.
[0077] In some possible implementations, the data model is ultimately constructed as a label integration layer. The structure of a data warehouse is generally a detail layer, a basic layer, and a service layer. The detail layer can provide a subject-oriented, integrated, current or nearly current, detailed data set, so that the subject details of business needs can be extracted based on the detail layer data, and a GUID is established for the subject details as an identifier to form a label detail layer; combining the definition of the label detail layer with the data application classification, the data is classified, aggregated, and converted into a bitmap to form a label integration layer, and the meaning information represented by the new label is included, refer to Figure 5 As shown, it is used to trace back the label detail data.
[0078] In some embodiments, the method provided by the embodiments of the present application further includes:
[0079] Create controls in data applications through front-end application plug-ins;
[0080] Through the front-end application plug-in, the circled nodes are converted into data exchange structures in the data application; the data exchange structure includes the circled tag name, logical relationship and tag value; the tag name is used to represent the tag table of the tag integration layer, and the tag value is used to represent the dimension value of the tag table.
[0081] In some embodiments, the method provided by the embodiment of the present application is to generate a data application based on the tag integration layer for the front-end application plug-in to interact with the event, including:
[0082] Expand the tag types based on the tag integration layer and establish interactive events;
[0083] Define the logical relationship of the labels based on the relationship in the bitmap function;
[0084] Add prompts and slogans to the interactive interface of the data application to improve the data application.
[0085] In some possible implementations, a user-selected data application is constructed based on the tag integration layer, referring to Figure 6 As shown, the front-end application react-awesome-query-builder plug-in is used for interactive events. The plug-in can provide input control creation, grouping control creation and JSON conversion of circled nodes, so as to establish a visual interaction of self-selection conditions. In addition, the plug-in needs to be further transformed to be suitable for the scene of tag circle selection. First, it is necessary to expand the interactive event system based on the tag type in the tag integration layer. The tag type is an enumeration type and needs to support drop-down selection and multiple selection; the text type supports text input; the number type supports number input and size judgment logic, and the time type supports start time and end time input and time before and after judgment logic; secondly, the logical relationship of and, or, and not of the tag needs to be defined, and the logic of not is realized by matching groups. Finally, there is a Chinese interactive interface, with supporting tag names, TIP prompts, optional logical judgment symbols and built-in slogans in the input box. After the user selects the conditions, a JSON structure representing the node data can be converted. The JSON body contains the selected labels, logical relationships and label values. The parsing rules are that the label name maps to the label table of the label integration layer, the logical relationship maps to the bitmap's AND, OR, and NOT functions, and the label value maps to the dimension value of the label table of the integration layer. Based on this rule and combined with the Trino SQL syntax, the query SQL is parsed and the SQL is executed to obtain business data through the selected conditions.
[0086] In some embodiments, reference Figure 7 As shown, the method provided in the embodiment of the present application further includes:
[0087] Step S410, classifying the groups based on the consumption data of the objects according to the preset cutoff value;
[0088] Step S420, creating a global unified identifier for each object, converting the classification result into a bitmap to construct a population model;
[0089] Step S430 , performing group cross operations on the objects in the group model according to the bitmap function and the object selection label, and determining the indicator data of each group.
[0090] In some possible implementations, user circled tags are combined to realize data application of crowd analysis, specifically, defining the dividing value of the crowd, dividing the customer into categories based on the three major factors of consumption time, frequency and amount, creating the customer's GUID, dividing the crowd based on the classification and converting it into a bitmap to build a crowd model, combining user circled tags with bitmap functions, grouping and cross-calculating to obtain the crowd in each category, and performing further statistical operations on the crowd to obtain the crowd's indicator data.
[0091] In some embodiments, the method provided by the embodiments of the present application further includes:
[0092] According to the tag name, logical relationship and tag value, the circled nodes are parsed, and the bitmap data set is determined through the cross operation of the bitmap function;
[0093] Determine a target global unified identifier according to the bitmap data set;
[0094] According to the target global unified identifier, the target subject details are determined based on the tag detail layer; the target subject details are the subject details of the target object.
[0095] In the embodiment of the present application, the data application of extracting marketing object details is realized based on the integration layer tag, referring to Figure 6 and Figure 8 As shown, based on the user's circled labels, label dimension values and logical relationships, node parsing and bitmap function are used to perform cross operations to obtain a new bitmap data set, from which the GUID of the marketing object is parsed. Based on the GUID, the detailed information of the marketing object is extracted by tracing back to the detail layer, thereby completing the rapid retrieval of the marketing object details and helping the business to carry out precision marketing.
[0096] The following is a detailed description and explanation of the solution of the embodiment of the present invention in conjunction with a specific application example:
[0097] It should be clear that the embodiments described this time are only a part of them, not all of them.
[0098] S41. Use the extensible features of bitmap in Hive and Trino to build custom bitmap AND, OR, and NOT logical functions and bitmap value conversion and bitmap aggregation statistical functions.
[0099] S42. Divide data based on business needs, construct a tag detail layer and a tag integration layer, establish a GUID in the detail layer, and load the tag data converted into a bitmap in the integration layer, and also include new tag information.
[0100] S43, based on the integration layer label information and the front-end plug-in to build interaction rules, combined with the definition of the RFM population boundary to build a population model (refer to Figure 8 ), through the grouping and cross-calculation of the bitmap function, we can get the population grouped by category, and further obtain the distribution of the RFM population, so as to understand the high-value and low-value populations and then carry out precision marketing to improve marketing effects.
[0101] The embodiment of the present application proposes a solution to establish a business label model through a bitmap combined with an extended bitmap function and front-end interaction, dynamically parse the Trino statement, and thus efficiently and quickly obtain marketing data to improve marketing effectiveness. The embodiment of the present application proposes a method to replace the manual writing of Trino query statements by label circling, thereby reducing time costs and improving application efficiency. The embodiment of the present application proposes a solution to support non-professionals in using big data technology, increasing the number of people using big data, improving data application efficiency, and at the same time reducing the error rate of Trino data queries, saving time and computing resources.
[0102] See also Fig. 9 The embodiment of the present application further provides a data processing system, which can implement the above data processing method, and the system includes:
[0103] The first module 810 is used to construct a bitmap function according to the extended support provided by Hive;
[0104] The second module 820 is used to obtain a data set related to business processing and construct a label integration layer based on the data set; the label integration layer is used to trace back label detail data;
[0105] The third module 830 is used to make interactive events for the front-end application plug-in and generate data applications based on the tag integration layer;
[0106] The fourth module 840 is used to receive the object selection tag based on the data application, determine the data query statement, and determine the business data according to the data query statement.
[0107] In some embodiments, the system provided by the embodiments of the present application further includes a sixth module for creating a control in a data application through a front-end application plug-in;
[0108] Through the front-end application plug-in, the circled nodes are converted into data exchange structures in the data application; the data exchange structure includes the circled tag name, logical relationship and tag value; the tag name is used to represent the tag table of the tag integration layer, and the tag value is used to represent the dimension value of the tag table.
[0109] In some embodiments, the system provided by the embodiments of the present application further includes a seventh module for classifying groups based on the consumption data of the objects according to a preset demarcation value;
[0110] Create a global unified identifier for each object and convert the classification results into a bitmap to build a population model;
[0111] According to the bitmap function and the object selection label, the objects in the group model are grouped and cross-calculated to determine the indicator data of each group.
[0112] In some embodiments, the system provided by the embodiments of the present application further includes an eighth module, which is used to parse the circled nodes according to the tag names, logical relationships and tag values, and determine the bitmap data set through the cross operation of the bitmap function;
[0113] Determine a target global unified identifier according to the bitmap data set;
[0114] According to the target global unified identifier, the target subject details are determined based on the tag detail layer; the target subject details are the subject details of the target object.
[0115] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0116] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0117] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0118] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0119] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0120] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the data processing method of the embodiment of this application;
[0121] Input / output interface 903, used to implement information input and output;
[0122] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0123] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0124] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0125] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data processing method when executed by a processor.
[0126] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0127] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0128] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0129] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0130] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0131] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0132] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0133] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0134] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0135] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0138] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A data processing method, characterized in that: The method comprises: Build bitmap functions based on the extended support provided by Hive; Acquire a data set related to the processing business, and build a label integration layer based on the data set; the label integration layer is used to trace back label detail data; Perform interactive events for the front-end application plug-in and generate data applications based on the tag integration layer; Based on the data application, receiving the object circled tag, determining a data query statement, and determining business data according to the data query statement; Obtain data sets related to the business being processed and build a label integration layer based on the data sets, including: Get the data set of the detail layer of the data warehouse; Based on the data set, determine the subject details of the business requirements; Establishing a global unified identifier for the subject details and constructing a tag detail layer; According to the label detail layer and data application classification, the data is classified and clustered to determine the label integration layer.
2. The method according to claim 1, characterized in that: The bitmap function is constructed according to the extended support provided by Hive, including: Create new classes; Using the first type of objects and the second type of objects as bitmap carriers, performing set operations on the first type of objects and the second type of objects, and determining a bitmap operation result set; The new class is encapsulated and registered with the Hive to construct a bitmap function.
3. The method according to claim 1, characterized in that The method further comprises: Creating a control in the data application through the front-end application plug-in; Through the front-end application plug-in, the selected nodes are converted into a data exchange structure in the data application; the data exchange structure includes the selected tag name, logical relationship and tag value; the tag name is used to represent the tag table of the tag integration layer, and the tag value is used to represent the dimension value of the tag table.
4. The method according to claim 1, characterized in that: The interactive event for the front-end application plug-in to generate a data application based on the tag integration layer includes: Expanding the tag type based on the tag integration layer to establish an interaction event; Defining the logical relationship of the tags according to the relationship in the bitmap function; Prompts and slogans are added to the interactive interface of the data application to improve the data application.
5. The method according to claim 4, characterized in that The method further comprises: According to the preset cut-off value, the group is classified based on the consumption data of the object; Create a global unified identifier for each object and convert the classification results into a bitmap to build a population model; According to the bitmap function and the object selection label, the objects in the group model are grouped and cross-calculated to determine the index data of each group.
6. The method according to claim 3, characterized in that The method further comprises: Parsing the circled nodes according to the tag name, the logical relationship and the tag value, and determining a bitmap data set through a cross operation of the bitmap function; Determining a target global unified identifier according to the bitmap data set; According to the target global unified identifier, the target subject details are determined based on the tag detail layer; the target subject details are subject details of the target object.
7. A data processing system, characterized in that: The system comprises: The first module is used to build a bitmap function based on the extended support provided by Hive; The second module is used to obtain a data set related to business processing and build a label integration layer based on the data set; the label integration layer is used to trace back label detail data; The third module is used to interact with the front-end application plug-in and generate a data application based on the tag integration layer; A fourth module is used to receive an object selection tag based on the data application, determine a data query statement, and determine business data according to the data query statement; The second module is used to: Obtain data sets related to the business being processed and build a label integration layer based on the data sets, including: Get the data set of the detail layer of the data warehouse; Based on the data set, determine the subject details of the business requirements; Establishing a global unified identifier for the subject details and constructing a tag detail layer; According to the label detail layer and data application classification, the data is classified and clustered to determine the label integration layer.
8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
