E-commerce big data-oriented data processing method and device, equipment and medium

Abstract

The invention discloses a data processing method and device for e-commerce big data, equipment and a medium, and relates to the technical field of big data. The method comprises the following steps: extracting e-commerce detail data from a target database based on a preset configuration rule, and forwarding the e-commerce detail data to message-oriented middleware for storage; the e-commerce detail data comprises business data, entity data, rule data or configuration data; processing the e-commerce detail data stored in the message-oriented middleware to obtain target processing results, and synchronizing the target processing results to a first database; establishing a materialized view structure comprising at least two layers based on each target processing result stored in the first database; the materialized view structure layers matched with the target processing results are determined respectively, the target processing results are displayed on the materialized view structure layers respectively, the e-commerce big data can be rapidly and accurately processed, complexity is reduced, and the processing results are effectively displayed.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device, equipment and medium for e-commerce big data. Background Art

[0002] With the booming development of e-commerce, the amount of data generated by e-commerce platforms is growing at an astonishing rate. This data contains a wealth of market information and user behavior. By collecting, organizing, and analyzing this data, e-commerce companies can gain a deeper understanding of market demand, consumer preferences, and competitor situations, thereby optimizing operational strategies and improving business efficiency and competitiveness.

[0003] At present, when processing e-commerce big data, data processing is often complicated and it is difficult to effectively display the processing results. How to process e-commerce big data quickly and accurately, reduce complexity and effectively display the processing results are key research issues in the industry. Summary of the Invention

[0004] The present invention provides a data processing method, device, equipment and medium for e-commerce big data, so as to quickly and accurately process e-commerce big data, reduce complexity and effectively display the processing results.

[0005] According to one aspect of the present invention, a data processing method for e-commerce big data is provided, the method comprising:

[0006] Extracting e-commerce detailed data from the target database based on preset configuration rules, and forwarding the e-commerce detailed data to the message middleware for storage; the e-commerce detailed data includes: business data, entity data, rule data or configuration data;

[0007] Processing each e-commerce detailed data stored in the message middleware to obtain a target processing result, and synchronizing each target processing result to the first database;

[0008] Establishing a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database;

[0009] Materialized view structure layers matching the target processing results are determined respectively, and the target processing results are displayed respectively on the materialized view structure layers.

[0010] According to another aspect of the present invention, a data processing device for e-commerce big data is provided, the device comprising:

[0011] A data extraction module is used to extract e-commerce detailed data from the target database based on preset configuration rules and forward the e-commerce detailed data to the message middleware for storage; the e-commerce detailed data includes: business data, entity data, rule data or configuration data;

[0012] The data processing module is used to process the detailed data of each e-commerce company stored in the message middleware to obtain target processing results, and synchronize each target processing result to the first database;

[0013] a materialized view structure establishing module, configured to establish a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database;

[0014] The display module is configured to respectively determine a materialized view structure layer that matches each target processing result, and display each target processing result in each materialized view structure layer.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data processing method for e-commerce big data described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method for e-commerce big data described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the data processing method for e-commerce big data described in any embodiment of the present invention.

[0021] The technical solution of the embodiment of the present invention extracts e-commerce detail data from the target database based on preset configuration rules, and forwards the e-commerce detail data to the message middleware for storage; the e-commerce detail data includes: business data, entity data, rule data or configuration data; each e-commerce detail data stored in the message middleware is processed to obtain a target processing result, and each target processing result is synchronized to the first database, so that real-time calculation of the detail data can be achieved; a materialized view structure containing at least two layers is established based on the attribute information of each target processing result stored in the first database; the materialized view structure layer matching each target processing result is determined respectively, and each target processing result is displayed in each materialized view structure layer respectively, which can quickly and accurately process e-commerce big data, reduce complexity and effectively display the processing results.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a flowchart of a data processing method for e-commerce big data provided in accordance with the first embodiment of the present invention;

[0025] Figure 2 This is a flowchart of a data processing method for e-commerce big data provided in accordance with the second embodiment of the present invention;

[0026] Figure 3 This is a flowchart of a data processing method for e-commerce big data provided in accordance with the third embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of a data processing system for e-commerce big data provided by Embodiment 3 of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of a data processing device for e-commerce big data provided by Embodiment 4 of the present invention;

[0029] Figure 6 It is a structural diagram of an electronic device that implements the data processing method for e-commerce big data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a data processing method for e-commerce big data provided in accordance with the first embodiment of the present invention. This embodiment is applicable to the case where a large amount of data generated by an e-commerce platform is processed in real time and effectively. The method can be executed by a data processing device for e-commerce big data. The data processing device for e-commerce big data can be implemented in the form of hardware and / or software. The data processing device for e-commerce big data can be configured in electronic devices such as computers, servers or tablet computers. Figure 1 As shown, the method includes:

[0034] Step 110: extract e-commerce detailed data from the target database based on preset configuration rules, and forward the e-commerce detailed data to the message middleware for storage.

[0035] The e-commerce detailed data includes: business data, entity data, rule data or configuration data.

[0036] Optionally, in an optional implementation of this embodiment, business data (for example, order data, sales data or after-sales data, etc.), entity data (for example, product name, product identification or product attributes, etc.), rule data (for example, classification standards, data sources or permission management, etc.) and configuration data (for example, payment method configuration, logistics and distribution settings, promotion activity configuration or inventory management, etc.) generated by the business system can be obtained in real time through the message middleware.

[0037] It should be noted that the detailed data extracted in this embodiment may include not only the detailed data generated by the e-commerce platform, but also the data generated by business systems such as the Enterprise Resource Planning (ERP) system, logistics management system or production management system, which is not limited in this embodiment.

[0038] In this embodiment, the preset configuration rules may include multiple configuration services, such as source data configuration service, extraction range configuration service, extraction frequency configuration service, incremental data monitoring service or extraction data storage service, etc., which are not limited in this embodiment.

[0039] It can be understood that in this embodiment, the source of the e-commerce detailed data to be extracted can be determined based on the source data configuration service in the preset configuration rules, for example, the order data or return data about merchant A or merchant B, etc.; the generation time of the e-commerce detailed data to be extracted can be determined based on the extraction range configuration service, for example, the order data or logistics data generated within the past three days can be extracted; the data extraction frequency can be determined based on the extraction frequency configuration service, for example, data can be extracted every five minutes or ten minutes, etc.; the incremental data generated by the e-commerce platform can be obtained based on the incremental data monitoring service, for example, newly submitted orders or logistics data, etc., which are not limited in this embodiment.

[0040] Optionally, in this embodiment, after each e-commerce detailed data is extracted based on each configuration service in the preset configuration rules, the extracted data can be further forwarded to the message middleware, and the extracted data can be stored through the message middleware; in this embodiment, the message middleware can be Java message middleware.

[0041] In an optional implementation of this embodiment, e-commerce detailed data can be directly captured from the target database and stored based on preset configuration rules through Java message middleware; in this embodiment, the target database can be a database management system such as SQL Server or MySQL, which is not limited in this embodiment.

[0042] Step 120: Process the detailed data of each e-commerce company stored in the message middleware to obtain a target processing result, and synchronize the target processing results to the first database.

[0043] The first database may be another distributed database other than the target database, which is not limited in this embodiment.

[0044] Optionally, in this embodiment, after the extracted e-commerce detail data is stored in the message middleware, the e-commerce detail data stored in the message middleware can be further processed to obtain the target processing result. Furthermore, the obtained target processing result can be synchronized to the first database.

[0045] In an optional implementation of this embodiment, the extracted e-commerce detail data can be processed in real time by a streaming computing engine and connected to the first database. Based on the table structure of the first database, the various parameters of the target processing results can be written (i.e., written into the first database) to achieve efficient and persistent storage of data.

[0046] Step 130: Establish a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database.

[0047] Among them, the attribute information of each target processing result can be a label indicating the target quantity result category, which can be used to determine whether the target processing result is original data, intermediate data or result data; that is, in this embodiment, the target processing results can be classified based on the attribute information of each target processing result, and further, the number of layers of the materialized view structure can be determined based on the classification result; exemplarily, if the target processing result contains original data and result data, then the number of layers of the materialized view structure can be two layers; if the target processing result contains original data, intermediate data and result data, then the number of layers of the materialized view structure can be three layers.

[0048] Furthermore, a materialized view structure may be created based on the determined number of layers; wherein each layer of the materialized view structure may be used to display different data.

[0049] Step 140 : Determine the materialized view structure layers that match the target processing results respectively, and display the target processing results respectively in the materialized view structure layers.

[0050] Optionally, in this embodiment, after creating a multi-layer materialized view structure, the materialized view structure layer that matches each target processing result can be determined separately, that is, the data category stored in each materialized view structure layer. Furthermore, each target processing result can be displayed separately in different materialized view structure layers.

[0051] In a specific example of this embodiment, the materialized view structure may include two layers of materialized view structure. The first layer of the materialized view structure may be used to store a real-time materialized view structure of order details. The first layer of the materialized view structure may be used to associate the order details fact table, the after-sales order fact table, the product dimension table, and the configuration table through SQL statements to generate wide table data, which may include the following:

[0052] Automatic SQL rewriting and optimization: Automatically rewrites user-submitted SQL into hierarchical calculation logic, splitting multi-table JOINs into intermediate steps that can be executed in parallel.

[0053] Batch initialization: When building for the first time, you can directly read the historical data of the base table for full batch processing, and it only takes a few minutes to initialize hundreds of millions of data points.

[0054] Incremental update: Based on the log of the base table, only incremental data changes are processed to reduce computing resource consumption;

[0055] Queryability of intermediate results: The results of intermediate calculation steps are stored in temporary tables and can be queried directly using SQL, facilitating debugging and data verification.

[0056] The second-layer materialized view structure can be used to store product-level aggregated results. Based on the first-layer real-time materialized view structure, aggregate calculations can be performed by product dimension, which may include the following:

[0057] Dynamic aggregation granularity: Aggregation can be performed by dimensions such as time, merchant, channel, and product, and business indicators can be generated using predefined aggregation functions (for example, SUM, COUNT, or AVG).

[0058] Incremental aggregation maintenance: When the first-layer materialized view structure data is updated, only the aggregation results of the relevant product dimensions can be incrementally updated to ensure real-time performance and low latency.

[0059] Drill-down consistency: Aggregation results are strongly associated with detailed data, allowing users to quickly drill down to the original order detail records through product dimension indicators, ensuring data traceability consistency.

[0060] The technical solution of this embodiment extracts e-commerce detailed data from the target database based on preset configuration rules, and forwards the e-commerce detailed data to the message middleware for storage; the e-commerce detailed data includes: business data, entity data, rule data or configuration data; each e-commerce detailed data stored in the message middleware is processed to obtain a target processing result, and each target processing result is synchronized to the first database, so that real-time calculation of the detailed data can be achieved; a materialized view structure containing at least two layers is established based on the attribute information of each target processing result stored in the first database; the materialized view structure layer matching each target processing result is determined respectively, and each target processing result is displayed in each materialized view structure layer respectively, which can quickly and accurately process e-commerce big data, reduce complexity and effectively display the processing results.

[0061] Example 2

[0062] Figure 2 This is a flow chart of a data processing method for e-commerce big data provided by the second embodiment of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0063] Step 210: extract e-commerce detailed data from the target database based on preset configuration rules, and forward the e-commerce detailed data to the message middleware for storage.

[0064] Optionally, in this embodiment, extracting e-commerce detail data from the target database based on preset configuration rules and forwarding the e-commerce detail data to the message middleware for storage may include: responding to processing instructions of different e-commerce businesses in real time, obtaining each business data generated during the processing of each e-commerce business, and storing each business data in the target partition of the message middleware; wherein the number of the target partitions is determined by the data volume of the business data; or, querying the rule data and configuration data stored in the reference database; storing each business data, each rule data and each configuration data in different partitions of the message middleware based on the attribute information of each business data, each rule data and each configuration data.

[0065] Among them, the e-commerce business may include reservation business, order business, after-sales business or insurance business, etc., which are not limited in this embodiment.

[0066] In an optional implementation of this embodiment, it is possible to respond to processing instructions of different e-commerce businesses in the e-commerce platform in real time, obtain various business data generated during the processing of each e-commerce business in real time, and store the generated business data in the target partition of the message middleware; in this embodiment, business data generated by the same e-commerce business can be stored in the same partition. When the amount of business data is large and cannot be stored in the same partition, it can be stored in the next partition.

[0067] In another optional implementation of this embodiment, the rule data and configuration data stored in the reference database can also be queried, wherein the reference database can be a database that temporarily stores the business data of various e-commerce businesses in the e-commerce platform. The type of this database is not limited in this embodiment.

[0068] Furthermore, the business data, rule data, and configuration data obtained above may be stored in different partitions of the message middleware.

[0069] In an optional implementation of this embodiment, the business data generated by each business during the processing process can be obtained in real time through the message middleware. For example, the business data such as the transaction amount business data, the order time, the identifier of the ordering user or the link information of the ordered goods in the order business can be obtained; further, the entity data associated with each of the business data can be queried, for example, the product name, product identifier or product attributes corresponding to each of the business data can be queried.

[0070] In another optional implementation of this embodiment, the rule data and configuration data associated with each business stored in the reference database can also be queried in real time through the message middleware; wherein, the reference database can be a relational database or a distributed database, etc., and the number can be one or more, which is not limited in this embodiment; in this embodiment, the rule data and configuration data can be stored in different databases, or in different areas of the same database; for example, the rule data can be stored in the first database, the configuration data of the first business can be stored in the first area of the second database, and the configuration data of the second business can be stored in the second area of the second data, etc., which is not limited in this embodiment.

[0071] In an optional implementation of this embodiment, after obtaining the detailed data involved in the above steps, each business data, each entity data, each rule data and each configuration data can be further stored in different detailed data tables of the target database based on the attribute information of the obtained business data, each entity data, each rule data and each configuration data.

[0072] The attribute information of each detailed data may include the category, generation time, business identifier of the detailed data, etc. The target database may also be a relational database or a distributed database, etc., which is not limited in this embodiment.

[0073] Step 220: Receive a real-time computing task; process each e-commerce detailed data based on the real-time computing task to obtain the target processing result.

[0074] Among them, the target processing result is a structured processing result.

[0075] In this embodiment, the computing task received in real time may be a profit analysis task, an inventory analysis task, or a purchasing task for a single commodity, which is not limited in this embodiment.

[0076] Optionally, in this embodiment, after storing the above-obtained e-commerce detailed data based on the message middleware, computing tasks can be received in real time. Furthermore, the e-commerce detailed data can be processed based on the task processing target of the received computing task to obtain the corresponding target processing result.

[0077] Optionally, in this embodiment, processing each of the e-commerce detail data based on the real-time computing task to obtain the target processing result may include: parsing the real-time computing task to determine the data cleaning rules, dimension expansion logic or cross-table association strategy embedded in the real-time computing task; cleaning each of the e-commerce detail data based on the data cleaning rules to obtain the cleaned first e-commerce detail data; or, associating each of the e-commerce detail data with a preset static dimension table based on the dimension expansion logic, supplementing the dimension information of each of the e-commerce detail data according to the association result to obtain the second e-commerce detail data; or determining the associated data of the target e-commerce detail data based on the cross-table association strategy, and associating the target e-commerce detail data with the associated data in real time to obtain the third e-commerce detail data; determining the target computing target of the real-time computing task, and determining the target processing model based on the target computing target; processing the first e-commerce detail data, the second e-commerce detail data or the third e-commerce detail data based on the target processing model to obtain the target processing result.

[0078] In an optional implementation of this embodiment, after receiving the real-time computing task, the real-time computing task can be further parsed to determine data rules such as data cleaning rules, dimension expansion logic or cross-table association strategies embedded in the real-time task.

[0079] Furthermore, the e-commerce detail data obtained above can be cleaned based on data cleaning rules to obtain the cleaned first e-commerce detail data; or, the e-commerce detail data can be associated with the preset static dimension table based on the dimension extension logic, and the missing dimension information of each e-commerce detail data can be determined according to the association result, and the dimension information of each e-commerce detail data can be supplemented to obtain the second e-commerce detail data; or, the associated data of the target e-commerce detail data can be determined based on the cross-table association strategy, and the target e-commerce detail data can be associated with the associated data in real time to obtain the third e-commerce detail data.

[0080] Furthermore, the calculation target of the real-time calculation task can be determined based on the analysis result of the real-time calculation task, and further, the target processing model can be determined based on the calculation target; for example, if the calculation target is the net profit analysis of the target product, then the target processing model can be the net profit analysis model; if the calculation target is the average net profit analysis of all products, then the target processing model can be the average net profit analysis model.

[0081] Furthermore, the first e-commerce detailed data, the second e-commerce detailed data or the third e-commerce detailed data determined above may be processed based on the target processing model respectively, thereby obtaining a target processing result.

[0082] Step 230: Acquire attribute information of each target processing result; determine a category of each target processing result based on the attribute information, determine a target number of layers of a materialized view structure based on the category; and establish the materialized view structure based on the target number of layers.

[0083] The attribute information includes: original data, intermediate data or result data;

[0084] Optionally, in this embodiment, after the target processing results are obtained through real-time calculation, it is possible to further determine whether each target processing result is original data, intermediate data, or result data; further, the number of layers of the materialized view structure can be determined based on the category of the target processing results, wherein the number of layers of the materialized view structure can be two, three, or four layers, etc., which is not limited in this embodiment.

[0085] Step 240: Filter the target processing result based on the identification information of the target materialized view structure layer to obtain a reference processing result; store and display the reference processing result in the target materialized view structure layer.

[0086] The target materialized view structure layer can be any layer of the materialized view structure, such as the first layer or the second layer, and is not limited in this embodiment. The identification information of the target materialized view structure layer is the label of the materialized view structure layer, which can be used to indicate the category information of the data stored in the layer.

[0087] In this embodiment, the reference processing result is the processing result that matches the identification information of the target materialized view structure layer among the target processing results.

[0088] Optionally, in this embodiment, after determining the number of layers of the materialized view structure, the target processing results can be further filtered based on the identification information of the target materialized view structure layer to obtain processing results that match the identification information of the target materialized view structure layer. Furthermore, the reference processing results obtained by filtering can be displayed in the target materialized view structure layer.

[0089] The solution of this embodiment can build a multi-level data aggregation model based on real-time materialized view technology, and can generate multi-dimensional analysis results through hierarchical calculations, providing a basis for supporting high-concurrency queries and dynamic indicator derivation.

[0090] Example 3

[0091] Figure 3 This is a flow chart of a data processing method for e-commerce big data provided by Example 3 of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 3 As shown, the method includes:

[0092] Step 310: extract e-commerce detailed data from the target database based on preset configuration rules, and forward the e-commerce detailed data to the message middleware for storage.

[0093] Step 320: Process the detailed data of each e-commerce company stored in the message middleware to obtain target processing results, and synchronize the target processing results to the first database.

[0094] Step 330: Establish a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database.

[0095] Step 340 : Determine the materialized view structure layers that match the target processing results, and display the target processing results in the materialized view structure layers.

[0096] Step 350: In response to the target calculation indicator change instruction, determine the target change information; re-execute the operation of determining the materialized view structure based on the target change information to obtain each new materialized view structure layer; and store the recalculation result based on the target calculation indicator in each of the new materialized view structure layers.

[0097] The target computing index may be a computing index of a received real-time computing task.

[0098] Optionally, in this embodiment, after receiving the change instruction of the target calculation indicator, specific change information can be determined, which is called target change information in this embodiment. Furthermore, the materialized view structure can be determined based on the target change information, for example, the number of layers of the materialized view structure, and the type of data stored in each layer, etc., so as to obtain new materialized view layers, that is, a new materialized view structure.

[0099] Furthermore, the recalculation results of the target calculation indicators may be stored in each of the new materialized view structure layers.

[0100] In an optional implementation of this embodiment, the materialized view structure supports full data recalculation services. When the business scope changes (for example, adding a dimension field or modifying an indicator formula), the following operations can be triggered: When the business scope changes (for example, adding a dimension field, modifying an indicator calculation formula, or adjusting the aggregation granularity), the materialized view structure can be redefined using the CREATE OR REPLACE INCREMENTAL MATERIALIZED VIEW statement. Full data recalculation is completed at the minute level, and subsequent incremental data is calculated based on the latest view structure.

[0101] The solution of this embodiment can ensure global data consistency when indicators are changed by fully recalculating the service.

[0102] In order to better understand the data processing for e-commerce big data involved in this embodiment, Figure 4 This is a structural diagram of a data processing system for e-commerce big data provided according to the third embodiment of the present invention, which mainly includes: a data extraction layer, a data preprocessing layer, a data synchronization layer and a materialized view layer.

[0103] The data extraction layer includes the source data source configuration service, extraction range configuration service, extraction frequency configuration service, incremental data monitoring service, and extracted data storage service. It is used to extract incremental data from multiple types of business databases according to configuration rules and transmit it to the message middleware.

[0104] Data preprocessing layer: This layer may include a real-time data processing module that uses a streaming computing engine to standardize incremental data, supplement dimension redundancy, and perform pre-correlation logic.

[0105] Data synchronization layer: This layer may include target data source configuration services, target table configuration services, and synchronization parameter control modules, and is used to write pre-processed data into the analytical database in real time.

[0106] Materialized view layer: This layer includes materialized view task generation services, incremental data processing services, and full data recalculation services. It implements basic data association, multi-dimensional aggregation, and dynamic indicator calculation through hierarchical views.

[0107] The solution of this embodiment can directly build real-time materialized views through SQL statements, simplifying the development process; the intermediate calculation process can be directly queried through SQL; batch initialization and incremental updates are supported, significantly shortening the cold start time; and global data consistency is ensured when indicators change through full recalculation services.

[0108] Example 4

[0109] Figure 5 This is a schematic diagram of the structure of a data processing device for e-commerce big data provided by the third embodiment of the present invention. Figure 5 As shown, the apparatus includes: a data extraction module 510 , a data processing module 520 , a materialized view structure establishment module 530 and a display module 540 .

[0110] The data extraction module 510 is configured to extract e-commerce detailed data from the target database based on preset configuration rules and forward the e-commerce detailed data to the message middleware for storage; the e-commerce detailed data includes: business data, entity data, rule data or configuration data;

[0111] The data processing module 520 is used to process the detailed data of each e-commerce company stored in the message middleware to obtain target processing results, and synchronize the target processing results to the first database;

[0112] A materialized view structure establishing module 530 is configured to establish a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database;

[0113] The display module 540 is configured to respectively determine a materialized view structure layer that matches each target processing result, and display each target processing result in each materialized view structure layer.

[0114] The solution of this embodiment is to extract e-commerce detailed data from the target database based on preset configuration rules through the data extraction module, and forward the e-commerce detailed data to the message middleware for storage; the data processing module processes the e-commerce detailed data stored in the message middleware to obtain target processing results, and synchronizes each target processing result to the first database; the materialized view structure establishment module establishes a materialized view structure containing at least two layers based on the attribute information of each target processing result stored in the first database; the display module determines the materialized view structure layer that matches each target processing result, and displays each target processing result in each materialized view structure layer. This can quickly and accurately process e-commerce big data, reduce complexity and effectively display the processing results.

[0115] In an optional implementation of this embodiment, the data extraction module 510 is specifically configured to:

[0116] Responding in real time to processing instructions of different e-commerce businesses, acquiring business data generated during the processing of each e-commerce business, and storing each business data in a target partition of the message middleware; wherein the number of the target partitions is determined by the amount of the business data;

[0117] Alternatively, query the rule data and configuration data stored in the reference database;

[0118] Based on the attribute information of each of the business data, each of the rule data and each of the configuration data, each of the business data, each of the rule data and each of the configuration data is stored in different partitions of the message middleware.

[0119] In an optional implementation of this embodiment, the data processing module 520 is specifically configured to:

[0120] Receive real-time computing tasks;

[0121] Based on the real-time computing task, each e-commerce detailed data is processed to obtain the target processing result; the target processing result is a structured processing result.

[0122] In an optional implementation of this embodiment, the materialized view structure establishing module 530 is specifically configured to:

[0123] Parsing the real-time computing task to determine the data cleaning rules, dimension expansion logic, or cross-table association strategy embedded in the real-time computing task;

[0124] Cleaning each e-commerce detailed data based on the data cleansing rules to obtain cleaned first e-commerce detailed data; or associating each e-commerce detailed data with a preset static dimension table based on the dimension expansion logic, supplementing each e-commerce detailed data with dimension information based on the association result to obtain second e-commerce detailed data; or determining associated data of target e-commerce detailed data based on a cross-table association strategy, and associating the target e-commerce detailed data with the associated data in real time to obtain third e-commerce detailed data;

[0125] determining a target computing target for the real-time computing task, and determining a target processing model based on the target computing target;

[0126] The first e-commerce detailed data, the second e-commerce detailed data or the third e-commerce detailed data is processed based on the target processing model to obtain the target processing result.

[0127] In an optional implementation of this embodiment, the materialized view structure establishing module 530 is further specifically configured to:

[0128] Acquiring attribute information of each target processing result; the attribute information includes: original data, intermediate data or result data;

[0129] determining a category of each target processing result based on the attribute information, and determining a target number of layers of a materialized view structure based on the category;

[0130] The materialized view structure is established based on the target number of levels.

[0131] In an optional implementation of this embodiment, the display module 540 is specifically used to

[0132] Filtering the target processing result based on the identification information of the target materialized view structure layer to obtain a reference processing result;

[0133] The reference processing result is stored and displayed in the target materialized view structure layer.

[0134] In an optional implementation of this embodiment, the data processing apparatus for e-commerce big data further includes a modification module configured to:

[0135] In response to a change instruction of a target calculation indicator, determining target change information;

[0136] Re-executing the operation of determining the materialized view structure based on the target change information to obtain each new materialized view structure layer;

[0137] The recalculation results based on the target calculation index are stored in each of the new materialized view structure layers.

[0138] The data processing device for e-commerce big data provided by the embodiment of the present invention can execute the data processing method for e-commerce big data provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0139] In the technical solution of the embodiment of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of various business data (such as order data, inventory data, etc.) of the e-commerce platform are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0140] Example 5

[0141] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0142] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0144] The processor 11 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a data processing method for e-commerce big data, the method comprising: extracting e-commerce detail data from a target database based on preset configuration rules, and forwarding the e-commerce detail data to a message middleware for storage; the e-commerce detail data includes: business data, entity data, rule data or configuration data; processing each e-commerce detail data stored in the message middleware to obtain a target processing result, and synchronizing each target processing result to a first database; establishing a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database; determining a materialized view structure layer that matches each target processing result, and displaying each target processing result in each materialized view structure layer.

[0145] In some embodiments, the data processing method for e-commerce big data can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method for e-commerce big data described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the data processing method for e-commerce big data in any other appropriate manner (for example, by means of firmware).

[0146] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes a messaging middleware component (e.g., an application server), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, messaging middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0151] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0152] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0153] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

[0154] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the database detection method provided in any embodiment of the present application.

[0155] The computer program product may be implemented by writing computer program code for performing the operations of the present invention in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0157] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method for e-commerce big data, characterized in that: include: Extract e-commerce detailed data from the target database based on preset configuration rules, and forward the e-commerce detailed data to the message middleware for storage; The e-commerce detailed data includes: business data, entity data, rule data or configuration data; Processing each e-commerce detailed data stored in the message middleware to obtain a target processing result, and synchronizing each target processing result to the first database; Establishing a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database; Materialized view structure layers matching the target processing results are determined respectively, and the target processing results are displayed respectively on the materialized view structure layers.

2. The data processing method for e-commerce big data according to claim 1 is characterized in that: The extracting e-commerce detailed data from the target database based on the preset configuration rules and forwarding the e-commerce detailed data to the message middleware for storage includes: Responding in real time to processing instructions of different e-commerce businesses, acquiring business data generated during the processing of each e-commerce business, and storing each business data in a target partition of the message middleware; wherein the number of the target partitions is determined by the amount of the business data; Alternatively, query the rule data and configuration data stored in the reference database; Based on the attribute information of each of the business data, each of the rule data and each of the configuration data, each of the business data, each of the rule data and each of the configuration data is stored in different partitions of the message middleware.

3. The data processing method for e-commerce big data according to claim 1, characterized in that: The pre-processing of each e-commerce detailed data stored in the message middleware to obtain the target processing result includes: Receive real-time computing tasks; Based on the real-time computing task, each e-commerce detailed data is processed to obtain the target processing result; the target processing result is a structured processing result.

4. The data processing method for e-commerce big data according to claim 3 is characterized in that: The processing of each e-commerce detailed data based on the real-time computing task to obtain the target processing result includes: Parsing the real-time computing task to determine the data cleaning rules, dimension expansion logic, or cross-table association strategy embedded in the real-time computing task; Cleaning each e-commerce detailed data based on the data cleansing rules to obtain cleaned first e-commerce detailed data; or associating each e-commerce detailed data with a preset static dimension table based on the dimension expansion logic, supplementing each e-commerce detailed data with dimension information based on the association result to obtain second e-commerce detailed data; or determining associated data of target e-commerce detailed data based on a cross-table association strategy, and associating the target e-commerce detailed data with the associated data in real time to obtain third e-commerce detailed data; Determining a target computing target for the real-time computing task, and determining a target processing model based on the target computing target; The first e-commerce detailed data, the second e-commerce detailed data or the third e-commerce detailed data is processed based on the target processing model to obtain the target processing result.

5. The data processing method for e-commerce big data according to claim 1 is characterized in that: The establishing of a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database includes: Acquiring attribute information of each target processing result; the attribute information includes: original data, intermediate data or result data; determining a category of each target processing result based on the attribute information, and determining a target number of layers of a materialized view structure based on the category; The materialized view structure is established based on the target number of levels.

6. The data processing method for e-commerce big data according to claim 5 is characterized in that: The determining of the materialized view structure layers that match the target processing results, and displaying the target processing results in the materialized view structure layers, respectively, includes: Filtering the target processing result based on the identification information of the target materialized view structure layer to obtain a reference processing result; The reference processing result is stored and displayed in the target materialized view structure layer.

7. The data processing method for e-commerce big data according to claim 1, characterized in that: The method further comprises: In response to a change instruction of a target calculation indicator, determining target change information; Re-executing the operation of determining the materialized view structure based on the target change information to obtain each new materialized view structure layer; The recalculation results based on the target calculation index are stored in each of the new materialized view structure layers.

8. A data processing device for e-commerce big data, characterized in that: include: A data extraction module is used to extract e-commerce detailed data from the target database based on preset configuration rules, and forward the e-commerce detailed data to the message middleware for storage; The e-commerce detailed data includes: business data, entity data, rule data or configuration data; The data processing module is used to process the detailed data of each e-commerce company stored in the message middleware to obtain target processing results, and synchronize the target processing results to the first database; a materialized view structure establishing module, configured to establish a materialized view structure comprising at least two layers based on the attribute information of each target processing result stored in the first database; The display module is configured to respectively determine a materialized view structure layer that matches each target processing result, and display each target processing result in each materialized view structure layer.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data processing method for e-commerce big data according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method for e-commerce big data according to any one of claims 1 to 7 when executed.

Images