Data processing method and device, electronic equipment and computer readable medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer readable medium, and relates to the technical field of computers.The specific implementation mode comprises the steps that wide table data and theme identification are obtained according to a received data processing request; obtaining corresponding state table data based on the wide table data, and generating target data based on the state table data, the wide table data and a preset filtering rule; determining a caliber calculation mode according to the target data and the theme identifier; calculating elements in the target data are extracted based on the caliber calculation mode for calculation, so that calculation result data are obtained; and updating the state table data according to the calculation result data. And the data processing speed and the universality of data processing logic are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, electronic device and computer-readable medium. Background Art

[0002] Currently, source data is produced by offline processing (spark), and data processing logic is implemented using columnar database structured query language. Data processing is slow to produce source data (at least T+1 day), and data processing logic is not universal. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a data processing method, device, electronic device and computer-readable medium, which can solve the problems of slow source data production and non-universal data processing logic in existing data processing.

[0004] To achieve the above objective, according to one aspect of an embodiment of the present application, a data processing method is provided, including:

[0005] Obtain wide table data and subject identifiers according to the received data processing request;

[0006] Based on the wide table data, the corresponding state table data is obtained, and based on the state table data, the wide table data and the preset filtering rules, the target data is generated;

[0007] Determine the caliber calculation method based on target data and subject identification;

[0008] Extract the calculation elements in the target data based on the caliber calculation method to perform calculations to obtain calculation result data;

[0009] Update the status table data according to the calculation result data.

[0010] Optionally, determine the caliber calculation method, including:

[0011] Determine the corresponding preset data type based on the subject identifier, judge whether the target data contains data corresponding to all the preset data types, and determine the caliber calculation method based on the judgment result data.

[0012] Optionally, the caliber calculation method is determined according to the judgment result data, including:

[0013] If the result data is determined to be data corresponding to all preset data types not contained in the target data, the primary key value corresponding to the target data is extracted, and the caliber calculation method is determined based on the primary key value.

[0014] Optionally, the caliber calculation method is determined based on the primary key value, including:

[0015] If the primary key value includes the number of orders, the caliber calculation method is determined as statistical submission of order data;

[0016] If the primary key value includes the sales amount, the calculation method is to use statistics of completed order data.

[0017] Optionally, the caliber calculation method is determined according to the judgment result data, including:

[0018] If the result data is determined to be data corresponding to all of the multiple preset data types contained in the target data, the caliber calculation method is determined to be statistics of submitted order data and statistics of completed order data.

[0019] Optionally, generate target data, including:

[0020] Extract the anti-duplicate key value corresponding to the wide table data in the status table data, and in response to the anti-duplicate key value being a first preset value, prompt duplicate processing information, and in response to the anti-duplicate key value being a second preset value, generate target data based on preset filtering rules and wide table data.

[0021] Optionally, generate target data, including:

[0022] The model conversion layer is called with the wide table data as input parameter to convert the wide table data into an object that can be recognized by the target configuration, thereby obtaining the converted wide table data;

[0023] When the state table data corresponds to the wide table data that has not been processed, the converted wide table data is filtered based on a preset filtering rule to generate target data.

[0024] In addition, the present application also provides a data processing device, including:

[0025] an acquisition unit, configured to acquire wide table data and a subject identifier according to a received data processing request;

[0026] A target data generating unit is configured to obtain corresponding state table data based on the wide table data, and generate target data based on the state table data, the wide table data and a preset filtering rule;

[0027] A caliber calculation method determination unit is configured to determine a caliber calculation method according to target data and a subject identifier;

[0028] A calculation unit is configured to extract calculation elements from the target data based on the caliber calculation method to perform calculations to obtain calculation result data;

[0029] The updating unit is configured to update the state table data according to the calculation result data.

[0030] Optionally, the caliber calculation method determination unit is further configured to:

[0031] Determine the corresponding preset data type based on the subject identifier, judge whether the target data contains data corresponding to all the preset data types, and determine the caliber calculation method based on the judgment result data.

[0032] Optionally, the caliber calculation method determination unit is further configured to:

[0033] If the result data is determined to be data corresponding to all preset data types not contained in the target data, the primary key value corresponding to the target data is extracted, and the caliber calculation method is determined based on the primary key value.

[0034] Optionally, the caliber calculation method determination unit is further configured to:

[0035] If the primary key value is configured as the number of orders, the caliber calculation method is determined as the statistical submission of order data;

[0036] If the primary key value is configured as sales amount, the calculation method is to count the completed order data.

[0037] Optionally, the caliber calculation method determination unit is further configured to:

[0038] If the result data is determined to be data corresponding to all of the multiple preset data types contained in the target data, the caliber calculation method is determined to be statistics of submitted order data and statistics of completed order data.

[0039] Optionally, the target data generating unit is further configured to:

[0040] Extract the anti-duplicate key value corresponding to the wide table data in the status table data, and in response to the anti-duplicate key value being a first preset value, prompt duplicate processing information, and in response to the anti-duplicate key value being a second preset value, generate target data based on preset filtering rules and wide table data.

[0041] Optionally, the target data generating unit is further configured to:

[0042] The model conversion layer is called with the wide table data as input parameter to convert the wide table data into an object that can be recognized by the target configuration, thereby obtaining the converted wide table data;

[0043] When the state table data corresponds to the wide table data that has not been processed, the converted wide table data is filtered based on a preset filtering rule to generate target data.

[0044] In addition, the present application also provides a data processing electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the data processing method as described above.

[0045] In addition, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the data processing method as described above is implemented.

[0046] To achieve the above objective, according to another aspect of the embodiments of the present application, a computer program product is provided.

[0047] A computer program product according to an embodiment of the present application includes a computer program, and when the program is executed by a processor, the data processing method provided by the embodiment of the present application is implemented.

[0048] An embodiment of the above invention has the following advantages or beneficial effects: the application obtains wide table data and subject identification according to the received data processing request; obtains corresponding state table data based on the wide table data, generates target data based on the state table data, wide table data and preset filtering rules; determines the caliber calculation method according to the target data and subject identification; extracts the calculation elements in the target data based on the caliber calculation method to calculate to obtain calculation result data; and updates the state table data according to the calculation result data. The data processing speed and the versatility of the data processing logic are improved.

[0049] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application.

[0051] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application;

[0052] Figure 2 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application;

[0053] Figure 3 is a structural diagram illustrating a data processing method according to an embodiment of the present application;

[0054] Figure 4 It is a main flow chart of a data processing method according to an embodiment of the present application;

[0055] Figure 5 is an example diagram of a data processing method according to an embodiment of the present application;

[0056] Figure 6 is a schematic diagram of main units of a data processing device according to an embodiment of the present application;

[0057] Figure 7 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0058] Figure 8 It is a structural diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following is an explanation of the exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of known functions and structures is omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of the present application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present application, certain software, components, models, and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or must have used the solution. In the technical solution of the present application, the collection, collection, update, analysis, processing, use, transmission, storage, etc. of the user personal information involved are in compliance with the provisions of relevant laws and regulations, are used for legal and reasonable purposes, and do not violate public order and good customs, are not shared, leaked or sold outside these legal uses, and are subject to supervision and management by regulatory authorities. Take necessary measures to prevent illegal access to such user personal information data, maintain the security of user personal information, network security and national security, ensure that persons who have access to personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once such user personal information data is no longer needed, the risk should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0060] When used, including in certain related applications, user privacy is protected by de-identifying data, such as by removing specific identifiers when used, controlling the amount or specificity of data stored, controlling how data is stored, and / or other methods of de-identification.

[0061] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the data processing method mainly includes the following steps S101 to S105.

[0062] Step S101, obtaining wide table data and subject identification according to the received data processing request.

[0063] In this embodiment, the execution subject of the data processing method (for example, it can be a server) can receive the data processing request through a wired connection or a wireless connection. When receiving the data processing request, the execution subject can obtain the wide table data of various formats corresponding to the request. The wide table data can be, for example, a wide table model, which refers to a database table in which indicators, dimensions, and attributes related to the business theme are associated, and obtain the subject identifier corresponding to the request. The subject identifier is used to characterize the type of subject involved in the data processing request of the embodiment of the present application. For example, the type of subject can include marketing subjects and transaction subjects, where the marketing subject focuses on placing an order without payment, and the transaction subject focuses on placing an order and paying.

[0064] Step S102, based on the wide table data, corresponding state table data is acquired, and based on the state table data, the wide table data and preset filtering rules, target data is generated.

[0065] Specifically, generating target data includes: extracting an anti-duplicate key value corresponding to the wide table data in the status table data, prompting duplicate processing information in response to the anti-duplicate key value being a first preset value, and generating target data based on preset filtering rules and the wide table data in response to the anti-duplicate key value being a second preset value.

[0066] The execution subject can extract the anti-duplicate key values ​​corresponding to each subject involved in the wide table data in the status table data, for example, the wide table primary key value, and in response to the anti-duplicate key value being a first preset value (for example, it can be 1, and the embodiment of the present application does not make specific restrictions on this, indicating that the corresponding wide table data has been processed), prompt duplicate processing information, and in response to the anti-duplicate key value being a second preset value (for example, it can be 0, and the embodiment of the present application does not make specific restrictions on this, indicating that the corresponding wide table data has not been processed), based on the preset filtering rules and wide table data, generate target data. The preset filtering rules can be "same department filtering", "cooperative third-party filtering", or "time filtering", and the embodiment of the present application does not make specific restrictions on the content of the filtering rules. When the anti-duplicate key value indicates that the corresponding wide table data has not been processed, the wide table data can be filtered based on the preset filtering rules to obtain the target data required for the next step of processing.

[0067] Step S103, determining the caliber calculation method according to the target data and the subject identifier.

[0068] The calculation basis may be, for example, a calculation method or a unit of measurement.

[0069] Different topic identifiers correspond to different caliber calculation methods. For example, when the topic identifier corresponds to a marketing topic, the caliber calculation method can be a calculation method based on the firepower value of the marketing scenario. The marketing firepower value can be used to measure the popularity of marketing products and user engagement. It integrates data from multiple dimensions such as the number of views, likes, comments, and shares of marketing products, reflecting the popularity of marketing products and user engagement. When the topic identifier corresponds to a transaction topic, the caliber calculation method can be to count the transaction order data (corresponding to placing an order and paying) or to count the submitted order data (corresponding to placing an order but not requiring payment). Among them, the transaction order data can include the order identifier and sales amount of the paid orders, and the submitted order data can include the order identifier of the submitted order but not paid, the order identifier of the submitted order and paid, the number of submitted orders that have been paid, and the number of submitted orders that have not been paid.

[0070] Step S104, extracting calculation elements in the target data based on the caliber calculation method to perform calculations to obtain calculation result data.

[0071] The calculation elements in the target data may include rule attributes, attribute types, target values, etc. For example, the rule attribute may be price, the attribute type may be interval, and the target value may be [50,80].

[0072] The calculation elements involved are extracted from the target data based on the caliber calculation method, and then the accumulation operation or reduction operation is performed based on the extracted calculation elements to accurately obtain the calculation result data. For example, the calculation result data can be the sales amount ranking data of a preset time period or the order quantity ranking data of a preset time period. The type of the calculation result data can be determined specifically according to the subject identifier. For example, if the subject identifier corresponds to a marketing subject, the calculation result data obtained can be the sales amount ranking data. If the subject identifier corresponds to a transaction subject, the calculation result data obtained can be a separate sales amount ranking data, a separate order quantity ranking data, or a data including sales amount ranking data and order quantity ranking data.

[0073] Step S105, updating the state table data according to the calculation result data.

[0074] After the calculation result data is obtained, it indicates that the corresponding wide table data has been processed, and the anti-duplicate key value of the state table data corresponding to the corresponding wide table data, that is, the primary key value, can be updated, for example, to 1, indicating that the corresponding wide table data has been successfully processed. The embodiment of the present application does not specifically limit this value. This avoids subsequent repeated data processing and ensures the efficiency and accuracy of data processing.

[0075] This embodiment obtains wide table data and subject identification according to the received data processing request; obtains corresponding state table data based on the wide table data, generates target data based on the state table data, wide table data and preset filtering rules; determines the caliber calculation method according to the target data and subject identification; extracts the calculation elements in the target data for calculation based on the caliber calculation method to obtain calculation result data; and updates the state table data according to the calculation result data. The data processing speed and the versatility of the data processing logic are improved.

[0076] Figure 2 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application, such as Figure 2 As shown, the data processing method mainly includes the following steps S201 to S205.

[0077] Step S201, obtaining wide table data and subject identification according to the received data processing request.

[0078] The wide table data can be different wide table models. The subject identifier can be a subject number or a subject name.

[0079] Step S202: based on the wide table data, obtain corresponding state table data, and generate target data based on the state table data, the wide table data and preset filtering rules.

[0080] The status table data can store the primary key of the wide table data, that is, the anti-duplicate key, and the corresponding primary key value, that is, the anti-duplicate key value. Data processing duplication verification is performed based on the anti-duplicate key value. When the verification passes (that is, the wide table data has not been processed before), the preset filtering rules are used to filter the wide table data to obtain the filtered target data.

[0081] Specifically, generating target data includes: calling a model conversion layer with wide table data as an input parameter to convert the wide table data into an object that can be recognized by the target configuration, thereby obtaining the converted wide table data; when the status table data corresponds to the wide table data that has not been processed, filtering the converted wide table data based on preset filtering rules, thereby generating target data.

[0082] Target configuration, for example, can be filtering rules and caliber calculation configuration. By converting wide table data into objects that can be recognized by the target configuration, configurability and universality can be achieved, thereby improving data processing efficiency.

[0083] Step S203, determining the corresponding preset data type according to the subject identifier, judging whether the target data contains data corresponding to all the preset data types, and determining the caliber calculation method according to the judgment result data.

[0084] The corresponding preset data type is determined according to the subject identifier. For example, if the subject identifier corresponds to a marketing subject, the obtained calculation result data may be sales amount sorting data, that is, the preset data type may be sales amount sorting data.

[0085] For example, if the subject identifier corresponds to the transaction subject, the calculated result data can be separate sales amount sorting data, separate order quantity sorting data, or a combination of sales amount sorting data and order quantity sorting data. The preset data type can be sales amount sorting data or order quantity sorting data or both sales amount sorting data and order quantity sorting data. For example, the preset data types corresponding to the transaction subject (e.g., multiple) are more than the preset data types of the marketing subject (e.g., one).

[0086] Specifically, the caliber calculation method is determined according to the judgment result data, including: if the judgment result data is data corresponding to the target data that does not contain all preset data types, extracting the primary key value corresponding to the target data, and determining the caliber calculation method according to the primary key value.

[0087] If the judgment result data is that the target data does not contain data corresponding to all preset data types, it can be determined that the target data corresponds to the transaction subject, and the primary key value corresponding to the target data can be extracted, such as one or more of the number of orders and sales amount, and the corresponding caliber calculation method can be determined based on the primary key value contained in the target data.

[0088] Specifically, the scope calculation method is determined according to the primary key value, including: if the primary key value includes the number of orders, the scope calculation method is determined to be statistical submission order data, and statistical submission order data corresponds to only placing an order without payment; if the primary key value includes the sales amount, the scope calculation method is determined to be statistical transaction order data, and statistical transaction order data corresponds to the need to place an order and pay. The scope calculation method to be used can be accurately determined according to the primary key value in the target data, thereby improving the accuracy of data processing.

[0089] Specifically, the caliber calculation method is determined according to the judgment result data, including: if the judgment result data is data corresponding to all multiple preset data types contained in the target data, then the caliber calculation method is determined to be statistics of submitted order data and statistics of completed order data.

[0090] If the judgment result data is data corresponding to all multiple preset data types contained in the target data, it can be determined that the target data corresponds to the transaction subject, and the calculation method is determined as counting the submitted order data (for example, counting the order IDs of orders that have been submitted but not paid, the order IDs of orders that have been submitted and paid, the number of submitted orders that have been paid and the number of submitted orders that have not been paid) and counting the completed order data (for example, counting the order IDs and sales amounts of paid and completed orders).

[0091] In some embodiments, the caliber calculation method is determined based on the judgment result data, including: if the judgment result data is data corresponding to a preset data type that is contained in all of the target data, it can be determined that the target data corresponds to a marketing theme, and the caliber calculation method can be determined to be a calculation method of the firepower value based on the marketing scenario (the marketing firepower value can be used to measure the popularity of marketing products and user participation. It combines data from multiple dimensions such as the number of views, likes, comments, and shares of marketing products, reflecting the popularity of marketing products and user participation). Thus, data processing is accurately performed based on the determined caliber calculation method.

[0092] Step S204, extracting calculation elements in the target data based on the caliber calculation method to perform calculations to obtain calculation result data.

[0093] The execution subject can extract the calculation elements in the target data based on the caliber calculation method. The calculation elements in the target data may include rule attributes, attribute types, target values, etc. For example, the rule attribute may be price, the attribute type may be an interval, and the target value may be [50,80]. The calculation result data obtained may be the top TOPn data obtained by sorting by the number of orders from most to least, or the top TOPn data obtained by sorting by the sales amount from most to least, where n is a positive integer. The embodiment of the present application does not specifically limit the value of n.

[0094] Step S205, updating the state table data according to the calculation result data.

[0095] Update the anti-duplicate key value in the status table data according to the calculation result data to facilitate subsequent repeated processing of data verification and ensure the efficiency and accuracy of data processing.

[0096] The terms that may be used in the embodiments of the present application are explained as follows: Fact Table: In a multidimensional data warehouse, a table that stores detailed values ​​of a metric or facts is called a "fact table", such as an order details table. Wide table model: usually refers to a database table that associates indicators, dimensions, and attributes related to a business subject. For example, the live broadcast wide table model is generated by associating the live broadcast basic information + live broadcast room transaction data + live broadcast room change explanation data. Star model: The star model is a multidimensional data relationship, which consists of a fact table (Fact table) and a set of dimension tables (Dimension Table). Dimension table: Dimension Table, which can be regarded as a window for users to analyze data, contains the characteristics of fact records in the fact data table. Usually, each dimension in the fact table has a dimension table, such as the goodsId field in the order details, which has a goods dimension table. Hbase is a distributed, column-oriented open source database.

[0097] Figure 3 FIG. 1 is a structural diagram illustrating a data processing method according to an embodiment of the present application. Figure 3 As shown, in the structure involved in the data processing method, the following have been built or connected: live broadcast and video at the user layer; messages and BDP tables at the external service layer; recall and sorting in general computing at the computing layer; MYSQL at the storage layer; message consumption at the data access layer; content flow data and content activity data from external data sources; rule management and permission management at the management background. Figure 3 The ones in bold are: service interface of external service layer; back calculation of computing layer; Hbase of storage layer; wide table merging of preprocessing layer; wide table model of model layer; one-click backup and fast recovery in disaster recovery. Subsequent construction or access includes: speed racing of user layer (that is, no payment is required after placing an order); recall plug-in and sorting plug-in in the secondary business development platform of computing layer; data supplement of preprocessing layer; batch pulling of data access layer; star model of model layer; order data of external data source; metadata management, routing isolation and expression engine of management background.

[0098] Figure 4 FIG. 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application. Figure 4 As shown, when receiving wide table data, the wide table data is input into the model conversion layer to convert different wide table data into objects that can be recognized by the filtering rules and caliber calculation configuration. For example, when executing the Flink list calculation task, wide table data, such as a wide table model:

[0099] Input wide table data (for example, it may include wide table of vision, RDDM wide table, live wide table task and other data) into the model conversion layer: the model conversion layer converts different wide table models into objects that can be recognized by filtering rules and caliber calculation configurations. Specifically: define a set of common models, including activities, sorting rules, filtering rules, and support many-to-many models, translate into ER models, and convert business models, such as live business, live activities are mapped to general activities, live tracks are mapped to sorting rules, and live filtering rules are mapped to filtering rules. Wide table access is configured, and metadata configuration describes the relationship between different models, the mapping and conversion between fields, etc. The configuration data is saved in the meta_model_relation_info table.

[0100] Extract the anti-duplicate key status judgment: extract the value (this calculated value) corresponding to the wide table primary key rowkey from the hbase status data. The value (this calculated value) represents the anti-duplicate key status. For example, if the value (this calculated value) is 1, it means repeated data processing. If the value (this calculated value) is 0, it means no processing. By extracting the anti-duplicate key status to determine whether the data is processed repeatedly, it can avoid wasting data processing and computing resources. For example, the hbase status data can be obtained from the hbase status table. For example, in the hbase status table, the transaction theme can be obtained: rowkey: wide table primary key (primary key extracted from model configuration)_version number, value: this score; you can also obtain the marketing theme: rowkey: collectorid_roomid_skuid_eventTime_versionid, value: 1.

[0101] Filter rule parsing and filtering: Define a set of general filter rule storage structures, which are independent of business attributes. For example, a rule is defined as an interval filter, an input box filter, a selection box filter, etc. For example, the structure is as follows:

[0102] "filter_code":"item_first_cate_cd", / / Rule code is globally unique

[0103] "filter_type":1002, / / Rule type, input box filter

[0104] "filter_name":"", / / Rule name

[0105] "filter_value_type":0, / / Rule value type 0-string

[0106] "contain_exclude":1 / / Whether to include 1-include 2-not include

[0107] Caliber calculation: By analyzing the caliber calculation configuration, the caliber calculation configuration is realized. Calculation caliber model:

[0108] Calibre rule meta-information extraction;

[0109] Calibre rule configuration (model abstraction);

[0110] Calculation formula = caliber rule * N + behavior rule;

[0111] Caliber rules: including rule attributes, attribute types, and target values;

[0112] Calibre behavior rules: including behavior type and behavior value, such as Figure 5 shown.

[0113] Figure 5 FIG. 1 is an example diagram of a data processing method according to an embodiment of the present application. Figure 5 As shown in the figure, for example, if the price is greater than 50 yuan and less than 80 yuan and the order has been paid, an accumulation operation of 1 is required. The calculation scope is broken down as follows: Rule 1: Price (price) is a rule attribute, the attribute type is interval, and the target value is [50,80]; Rule 2: Payment (isPay) is a rule attribute, the attribute type is boolean, and the target value is true (1); Behavior: The behavior type is accumulation, and the behavior value is 1.

[0114] Write the calculation result data of the caliber calculation into the calculation details, that is, write the hbase calculation detail data, for example, rowkey: list identifier + list element, value: {element value}, and then write the calculation result after sorting through the sorting layer, that is, write the hbase calculation result data, for example, rowkey: list identifier, value: {a:100, b:80, c:60}, and finally update the write status value to facilitate the subsequent accurate anti-duplicate key status judgment based on the status value written this time, so as to ensure the efficiency and accuracy of data processing. For example, writing hbase calculation detail data can be writing the calculation detail data into the hbase pound cake detail table, which can be: under the transaction theme, rowkey: [caliber type]_[calculation type]_[time granularity]_[sort dimension]_[rule identifier]_[version number]_[list element id], value: the total score of the list element; under the marketing theme, rowkey: [activity id]_[scenario id]_[granularity time]_[scenario stakeholder id]_[version number], value: single score. For example, writing hbase calculation result data can be writing the calculation result data into the hbase list result table, which can be: under the transaction theme, rowkey: [caliber type]_[calculation type]_[time granularity]_[sort dimension]_[rule identifier]_[version number], value: [element a: 100, element b: 80...]; under the marketing theme, rowkey: [activity id]_[scenario id]_[granularity time]_[version number], value: [scenario stakeholder id: total score...].

[0115] The calculation result data of hbase can be provided to the outside world through the JSF framework, message queue MQ, and data warehouse tool HIVE to allow users to view the corresponding list.

[0116] It is understandable that in the metadata configuration background, the topic and conversion rules can be configured through the data source management to configure the primary key (for example, it can be an anti-duplicate key) for anti-duplicate key status judgment; for example, the configured topic and conversion rules can be, for example:

[0117] Under the marketing theme:

[0118] Wide table access (configuration): topic:live_theme_1

[0119] clientid:b4565c56

[0120] Model field conversion rules (configuration):

[0121] Original field->target field, type, whether it is the primary key...

[0122] trackid->sceneid, long, no

[0123] liveid->roomid, long, yes

[0124] skuid->skuid, long, yes

[0125] …

[0126] Wide table primary key: roomid_skuid.

[0127] You can configure filtering rules through filtering rule management to facilitate accurate rule filtering; for example, the filtering rule configuration can be as follows:

[0128] General filtering rule configuration (model abstraction): {

[0129] "filter_code": "thirdTag", / / Rule code is globally unique

[0130] "filter_type":1001, / / Rule type, text type 1001

[0131] "filter_name": "", / / Rule name

[0132] "filter_value_type": 0, / / Rule value type 0-integer

[0133] "contain_exclude": 1, / / Whether to include 1-include 2-not include

[0134] …

[0135] }

[0136] You can configure the calculation caliber through calculation rule management to facilitate accurate caliber calculation; for example, the calculation caliber configuration can be as follows:

[0137] Calculation rule configuration (model abstraction):

[0138] Calculation formula = rules * N + behavior

[0139] Rules: Extract rule metadata, including rule attributes, attribute types, and target values

[0140] Behavior: including behavior type and behavior value

[0141] For example, if the price is greater than 50 yuan and less than 80 yuan and the order has been paid, an accumulation operation rule 1 is required: price is the rule attribute, the attribute type is interval, and the target value is [50,80]

[0142] Rule 2: payment is a rule attribute, the attribute type is boolean, and the target is true(1)

[0143] Behavior: The behavior type is cumulative and the behavior value is 1.

[0144] Sorting rules can be configured through sorting rule management to facilitate sorting of calculation detail data through the sorting layer; for example, sorting rule management can be as follows:

[0145] General sorting rule configuration (model abstraction):

[0146] {

[0147] "sort_code":"score", / / sort code is globally unique

[0148] "sort_type": "", / / sorting method 0-heap sorting

[0149] "sort_name": "", / / sort name

[0150] "weight":20, / / weight

[0151] …

[0152] }

[0153] This technical solution is based on in-memory computing. Through model conversion, different wide table models are converted into objects that can be recognized by screening rules and caliber calculation configurations, thus achieving configurability; screening rules abstract rule types that are not related to business rules, thus achieving versatility; caliber calculation is the ultimate goal, and caliber rules*N+behavior rules are abstracted, thus achieving configurability. The embodiment of this application can shorten the source data of data processing production from days to minutes, thus achieving quasi-real-time production; it supports multi-dimensional screening, and realizes model conversion configurability, screening rule configurability, and caliber calculation configurability. It is mainly used in content ranking scenarios such as racing, live broadcast, and video, and can also be applied to marketing product selection scenarios, etc.

[0154] Figure 6 Schematic diagram of the main units of the data processing device according to an embodiment of the present application. Figure 6 As shown, the data processing device 600 includes an acquisition unit 601, a target data generation unit 602, an aperture calculation method determination unit 603, a calculation unit 604 and an update unit 605.

[0155] The acquisition unit 601 is configured to acquire wide table data and a subject identifier according to a received data processing request.

[0156] The target data generating unit 602 is configured to obtain corresponding state table data based on the wide table data, and generate target data based on the state table data, the wide table data and a preset filtering rule.

[0157] The caliber calculation method determination unit 603 is configured to determine the caliber calculation method according to the target data and the subject identifier.

[0158] The calculation unit 604 is configured to extract the calculation elements in the target data based on the caliber calculation method to perform calculations to obtain calculation result data.

[0159] The updating unit 605 is configured to update the state table data according to the calculation result data.

[0160] In some embodiments, the caliber calculation method determination unit 603 is further configured to: determine the corresponding preset data type according to the subject identifier, determine whether the target data contains data corresponding to all the preset data types, and determine the caliber calculation method according to the judgment result data.

[0161] In some embodiments, the caliber calculation method determination unit 603 is further configured to: if the judgment result data is that the target data does not contain data corresponding to all preset data types, extract the primary key value corresponding to the target data, and determine the caliber calculation method according to the primary key value.

[0162] In some embodiments, the caliber calculation method determination unit 603 is further configured as: if the primary key value is configured as the number of orders, the caliber calculation method is determined to be statistics of submitted order data; if the primary key value is configured as the sales amount, the caliber calculation method is determined to be statistics of completed order data.

[0163] In some embodiments, the caliber calculation method determination unit 603 is further configured to: if the judgment result data is data corresponding to all multiple preset data types contained in the target data, then the caliber calculation method is determined to be statistics of submitted order data and statistics of completed order data.

[0164] In some embodiments, the target data generation unit 602 is further configured to: extract the anti-duplicate key value corresponding to the wide table data in the status table data, prompt duplicate processing information in response to the anti-duplicate key value being a first preset value, and generate target data based on preset filtering rules and wide table data in response to the anti-duplicate key value being a second preset value.

[0165] In some embodiments, the target data generation unit 602 is further configured to: call the model conversion layer with the wide table data as an input parameter to convert the wide table data into an object that can be recognized by the target configuration, thereby obtaining the converted wide table data; when the status table data corresponds to the wide table data that has not been processed, filter the converted wide table data based on preset filtering rules to generate the target data.

[0166] It should be noted that the data processing method and the data processing device of the present application have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.

[0167] Figure 7 An exemplary system architecture 700 to which the data processing method or data processing device according to the embodiment of the present application can be applied is shown.

[0168] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0169] Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0170] The terminal devices 701 , 702 , and 703 may be various electronic devices having a data processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0171] Server 705 may be a server that provides various services, such as a background management server that supports data processing requests submitted by users using terminal devices 701, 702, and 703 (only as an example). The background management server may obtain wide table data and subject identification according to the received data processing request; obtain corresponding state table data based on the wide table data, and generate target data based on the state table data, wide table data, and preset filtering rules; determine the caliber calculation method according to the target data and subject identification; extract the calculation elements in the target data based on the caliber calculation method to calculate to obtain the calculation result data; and update the state table data according to the calculation result data. The data processing speed and the versatility of the data processing logic are improved.

[0172] It should be noted that the data processing method provided in the embodiment of the present application is generally executed by the server 705 , and accordingly, the data processing device is generally set in the server 705 .

[0173] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0174] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present application. Figure 8 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0175] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the computer system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0176] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0177] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present application are executed.

[0178] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0179] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0180] The units involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an acquisition unit, a target data generation unit, a caliber calculation method determination unit, a calculation unit, and an update unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0181] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device obtains wide table data and subject identification according to the received data processing request; based on the wide table data, obtains the corresponding state table data, and generates target data based on the state table data, the wide table data and the preset filtering rules; determines the caliber calculation method according to the target data and the subject identification; extracts the calculation elements in the target data based on the caliber calculation method for calculation to obtain the calculation result data; and updates the state table data according to the calculation result data.

[0182] The computer program product of the present application includes a computer program, and the computer program implements the data processing method in the embodiment of the present application when executed by a processor.

[0183] According to the technical solution of the embodiment of the present application, the data processing speed and the versatility of the data processing logic can be improved.

[0184] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data processing method, characterized in that: include: Obtain wide table data and subject identifiers according to the received data processing request; Based on the wide table data, corresponding state table data is acquired, and based on the state table data, the wide table data and a preset filtering rule, target data is generated; Determining a caliber calculation method based on the target data and the subject identifier; Extracting calculation elements from the target data based on the caliber calculation method to perform calculations to obtain calculation result data; The state table data is updated according to the calculation result data.

2. The method according to claim 1, characterized in that The method for determining the caliber calculation includes: Determine the corresponding preset data type according to the subject identifier, judge whether the target data contains all the data corresponding to the preset data type, and determine the caliber calculation method according to the judgment result data.

3. The method according to claim 2, characterized in that The method of determining the caliber calculation method according to the judgment result data includes: If the judgment result data is data corresponding to all of the preset data types not contained in the target data, the primary key value corresponding to the target data is extracted, and the caliber calculation method is determined according to the primary key value.

4. The method according to claim 3, characterized in that The method of determining the caliber calculation method according to the primary key value includes: If the primary key value includes the number of orders, the caliber calculation method is determined to be statistical submission of order data; If the primary key value includes the sales amount, the calculation method is determined to be statistical transaction order data.

5. The method according to claim 2, characterized in that: The method of determining the caliber calculation method according to the judgment result data includes: If the judgment result data is data corresponding to all the multiple preset data types contained in the target data, the caliber calculation method is determined to be statistics of submitted order data and statistics of completed order data.

6. The method according to claim 1, characterized in that The generating target data comprises: Extract the anti-duplication key value corresponding to the wide table data in the status table data, and in response to the anti-duplication key value being a first preset value, prompt duplicate processing information; in response to the anti-duplication key value being a second preset value, generate target data based on preset filtering rules and the wide table data.

7. The method according to claim 1, characterized in that The generating target data comprises: The model conversion layer is called with the wide table data as an input parameter to convert the wide table data into an object that can be recognized by the target configuration, thereby obtaining the converted wide table data; When the state table data corresponds to the wide table data and has not been processed, the converted wide table data is filtered based on a preset filtering rule to generate target data.

8. A data processing device, characterized in that: include: an acquisition unit, configured to acquire wide table data and a subject identifier according to a received data processing request; a target data generating unit configured to obtain corresponding state table data based on the wide table data, and generate target data based on the state table data, the wide table data and a preset filtering rule; a caliber calculation method determination unit, configured to determine a caliber calculation method according to the target data and the subject identifier; A calculation unit, configured to extract calculation elements in the target data based on the caliber calculation method to perform calculations to obtain calculation result data; An updating unit is configured to update the state table data according to the calculation result data.

9. A data processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images