A data processing method based on hybrid architecture collaboration and related equipment

Abstract

The application discloses a data processing method based on mixed architecture cooperation and related equipment, and the method comprises the following steps: extracting business data from an upstream business database according to a preset time point, performing data aggregation and index calculation processing on the business data based on a business rule through an analytical database to obtain a data model; synchronizing the data model to a relational database through a data synchronization tool, and generating an initial data view on an application interface based on the relational database; generating a query statement to query the relational database in response to an interactive instruction of the initial data view to obtain a query result; and performing rendering update processing on the initial data view according to the query result to obtain a target data view. The application embodiment can improve the processing efficiency of data and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and related equipment based on a hybrid architecture collaborative approach. Background Technology

[0002] Among related technologies, there is a direct data warehouse connection approach, which involves building a data warehouse based on a relational database and then directly connecting to the database through an interface via a management dashboard to execute queries. However, in practical applications, it has been found that when faced with complex multi-table joins and large-scale data aggregation queries, the system response time is often too long, making it difficult to meet the second-level response standard required by interactive analysis. Secondly, the two workloads of batch data loading (ETL) and online analytical processing (OLAP) queries are completely coupled, sharing the same set of computing and storage resources. This leads to a decrease in query performance or even service timeouts during ETL job execution, severely limiting the system's available time window and resulting in low data processing efficiency.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a data processing method and related equipment based on a hybrid architecture collaboration, which can improve the efficiency of data processing.

[0005] To achieve the above objectives, one aspect of this application proposes a data processing method based on a hybrid architecture collaboration, the method comprising: Business data is extracted from the upstream business database according to a preset time point. The business data is then aggregated and indexed by an analytical database based on business rules to obtain a data model. The data model is synchronized to a relational database using a data synchronization tool, and an initial data view is generated in the application interface based on the relational database. In response to the interactive command on the initial data view, a query statement is generated to perform query processing on the relational database and obtain query results; The initial data view is rendered and updated based on the query results to obtain the target data view.

[0006] In some embodiments, the step of aggregating and calculating metrics on the business data using an analytical database based on business rules to obtain a data model includes the following steps: The business data is divided into themes according to the business rules to obtain business themes; The business data is initially aggregated based on the business theme to obtain theme data. Based on the business rules, the business objects are determined, and cross-topic data extraction and aggregation are performed on the topic data based on the business objects to obtain the indicator summary data. The data model is obtained by performing multi-dimensional correlation and aggregation processing on the summarized data of the indicators.

[0007] In some embodiments, the initial aggregation processing of the business data according to the business theme to obtain theme data includes the following steps: Based on the business theme, determine the business primary key and relationships of the data tables to obtain the associated data tables; The related data tables are merged vertically to obtain the topic data.

[0008] In some embodiments, the step of performing cross-topic data extraction and aggregation on the topic data based on the business object to obtain summary indicator data includes the following steps: Based on the business object, the indicator data for different themes in the theme data is extracted to obtain the indicator data; The index data is aggregated and summarized according to different periods and different granularities to obtain the index summary data.

[0009] In some embodiments, synchronizing the data model to a relational database using a data synchronization tool and generating an initial data view in the application interface based on the relational database includes the following steps: The data synchronization tool initiates a loading task to fully synchronize the data model to a relational database; the relational database stores the historical data of the data model. Obtain the front-end configuration of the application interface, and initiate a data query request to the relational database based on the front-end configuration; The data query request is processed by the relational database to obtain the query data and generate the corresponding initial data view.

[0010] In some embodiments, generating a query statement in response to an interactive instruction on the initial data view to query the relational database and obtain query results includes the following steps: The interaction instructions are captured through the front-end framework of the application interface; the interaction instructions are used to adjust the filtering conditions of the initial data view. Based on the interactive instructions, the filtering conditions are determined, and the statement is constructed to obtain the query statement. The relational database is queried according to the query statement to obtain the query result.

[0011] In some embodiments, the step of determining the filtering conditions based on the interactive instructions and constructing the query statement includes the following steps: The interaction instruction is subjected to intent recognition to obtain the interaction intent, and the interaction intent is converted into the filtering conditions; The query statement is obtained by matching the interaction intent and passing the filtering conditions as parameters to the statement template for concatenation.

[0012] To achieve the above objectives, another aspect of this application proposes a data processing apparatus based on a hybrid architecture collaboration, the apparatus comprising: The data acquisition and calculation module is used to extract business data from the upstream business database according to a preset time point, and to perform data aggregation and indicator calculation processing on the business data based on business rules through an analytical database to obtain a data model. The data synchronization module is used to synchronize the data model to a relational database through a data synchronization tool, and generate an initial data view on the application interface based on the relational database. The query service module is used to generate a query statement in response to the interactive command of the initial data view to perform query processing on the relational database and obtain query results; The application display module is used to render and update the initial data view based on the query results to obtain the target data view.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a data processing method and related equipment based on a hybrid architecture collaboration. This solution extracts business data from an upstream business database according to a preset time point. An analytical database aggregates and calculates metrics based on business rules to obtain a data model. This allows for data aggregation based on business rules, reducing the overhead of data correlation during subsequent queries and improving data processing efficiency. Furthermore, this solution synchronizes the data model to a relational database using a data synchronization tool and generates an initial data view on the application interface based on the relational database, enabling efficient and complete data synchronization to the relational database. Additionally, this solution generates query statements in response to interactive commands on the initial data view, processes the query results in the relational database, and renders and updates the initial data view based on the query results to obtain the target data view. This isolates high-throughput batch write loads from low-latency random read loads from physical resources, ensuring the stability of the overall system performance and improving data processing efficiency. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of an implementation environment provided in the embodiments of this application; Figure 2 This is a flowchart of a data processing method based on a hybrid architecture provided in an embodiment of this application; Figure 3 This is a schematic diagram of an interface for managing the cockpit provided in an embodiment of this application; Figure 4 This is a sub-diagram of a commodity holding structure provided in an embodiment of this application; Figure 5 This is a schematic diagram of a hybrid architecture system provided in an embodiment of this application; Figure 6 This is a data processing flowchart provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a data processing device based on a hybrid architecture provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0022] 1) ETL (Extract, Transform, Load) is a process used for data warehousing and data integration, mainly including the following steps: Extract: Obtain raw data from various data sources (such as relational databases, files, APIs, etc.). Transform: Clean and format the extracted data to facilitate subsequent analysis. Load: Load the transformed data into the target data warehouse or database.

[0023] 2) A dashboard typically refers to a real-time monitoring and analysis platform that integrates key business data. It displays the overall operational status of a business through intuitive charts and metrics, helping managers quickly grasp the operational status and make informed decisions. Applications of digital dashboards include company operations management, user behavior analysis, and marketing effectiveness evaluation. Through data visualization technology, the dashboard can transform complex data into easily understandable information, supporting decision-makers in maintaining the right direction in complex business environments.

[0024] As enterprise digital transformation enters its more complex phase, data is no longer simply a record of operations but has evolved into a core strategic asset. This process necessitates a fundamental shift in the decision-making models of senior managers. They need to move from relying on periodic, outdated written reports to a dynamic system capable of instantly sensing business pulse and discerning trends. The management dashboard has emerged to meet this need, acting as the enterprise's "nerve center" and "strategic dashboard." Currently, several technical solutions exist for providing data to the management dashboard; however, these solutions all exhibit limitations when dealing with large datasets and high-concurrency scenarios.

[0025] Among related technologies, there is a traditional direct connection approach to data warehouses. This solution builds a data mart based on relational databases such as Oracle and Teradata or early MPP architectures. The technical implementation involves extracting, transforming, and loading business data into a normalized or multidimensional model of the warehouse. Subsequently, the management dashboard frontend directly connects to the data warehouse via JDBC or ODBC interfaces to execute queries. However, this solution has several shortcomings: First, when facing complex multi-table joins and large-scale data aggregation queries, the system response time often exceeds 10 seconds or even reaches the minute level, making it difficult to meet the second-level response standard required for interactive analysis. Second, the two workloads—Electronic Data Transfer (ETL) and Online Analytical Processing (OLAP) queries—are completely coupled, sharing the same set of computing and storage resources. This leads to a decrease in query performance or even service timeouts during ETL job execution, severely limiting the system's available time window.

[0026] Another evolutionary approach is to adopt a modern big data platform as a unified data processing and querying foundation. These platforms are typically built on the Hadoop ecosystem (such as Hive and Spark SQL) or next-generation MPP analytical databases (such as ClickHouse, Doris, and StarRocks). After data is processed within this platform, the dashboard application directly queries the result tables on the platform. While these platforms perform well in single complex queries, a single query or background task with abnormally high resource consumption can easily slow down the entire platform, impacting the dashboard experience for all users and posing a challenge to system concurrency.

[0027] In view of this, this application provides a data processing method and related equipment based on a hybrid architecture collaboration. This method extracts business data from an upstream business database according to a preset time point. An analytical database performs data aggregation and indicator calculation on the business data based on business rules to obtain a data model. A data synchronization tool synchronizes the data model to a relational database, and an initial data view is generated on the application interface based on the relational database. A query statement is generated in response to the interactive command on the initial data view to query the relational database and obtain the query results. The initial data view is then rendered and updated based on the query results to obtain the target data view. This method decouples the three core components of the data processing flow: "computation," "synchronization," and "service." This ensures data consistency and historical traceability while enabling collaborative processing of complex data, high-concurrency queries, and flexible exploration, thus building a stable and efficient data supply paradigm for the management dashboard.

[0028] This application embodiment achieves load isolation by introducing a KunDB database as a dedicated query layer. This allows the backend ArgoDB database to perform a large number of computational tasks, while the frontend KunDB can focus on providing services for dynamic real-time queries. This division of labor avoids resource contention that occurs when a single system simultaneously handles batch writes and random reads, resulting in more stable and efficient performance overall. Furthermore, the strategy of full-width table synchronization not only ensures the traceability of historical data but also enables any chart on the frontend to be converted into a single-table query, improving query performance at the data model level. In addition, the architecture of this application embodiment greatly enhances the interactive experience; its stable, second-level response capability allows the frontend to provide independent, concurrent interactive data processing capabilities for multiple views.

[0029] This application provides a data processing method based on a hybrid architecture collaboration, relating to the field of data processing technology. This hybrid architecture collaborative data processing method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a hybrid architecture collaborative data processing method, but is not limited to the above forms.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0032] Figure 1 This is a schematic diagram illustrating the implementation environment of a method provided in an embodiment of this application. (Refer to...) Figure 1 The main hardware and software components of this implementation environment include a terminal 101 and a server 102, which are communicatively connected. The method can be executed based on the interaction between the terminal 101 and the server 102. Furthermore, the terminal 101 and the server 102 can be nodes in a blockchain; this embodiment does not specifically limit this.

[0033] Figure 2 This is an optional flowchart of a data processing method based on a hybrid architecture provided in an embodiment of this application. Figure 2 The method may include, but is not limited to, steps S201 to S204.

[0034] Step S201: Extract business data from the upstream business database according to a preset time point, and perform data aggregation and indicator calculation processing on the business data based on business rules through an analytical database to obtain a data model; Step S202: Synchronize the data model to a relational database using a data synchronization tool, and generate an initial data view on the application interface based on the relational database; Step S203: In response to the interactive instruction on the initial data view, a query statement is generated to perform query processing on the relational database to obtain the query result; Step S204: Render and update the initial data view according to the query results to obtain the target data view.

[0035] Steps S201 to S204 of this embodiment involve extracting raw data from various upstream business databases and executing a complete ETL process, including data cleaning, multi-table joins, complex aggregations, and indicator calculations, in an analytical database such as ArgoDB. This ultimately generates a unified data model oriented towards the analytical topic. This data model can adopt a wide table structure, integrating key dimensions and indicators from different business domains such as market data, market trends, and trading into a single logical table, reducing table join overhead in subsequent queries. Furthermore, after daily processing of data collection and calculation, a data synchronization task is initiated to synchronize the generated data model to a relational database. This embodiment uses a relational database as a query engine, allowing for pre-querying of data based on the application interface and generating a corresponding initial data view. The relational database can be KunDB, which offers faster query speeds and supports higher concurrency. This embodiment also receives interactive commands for the initial data view in the application interface, efficiently converting user interaction intent into underlying query requests. Data retrieved from the relational database is then used to update the view, providing a smooth interactive experience.

[0036] One of the above technical solutions has the following advantages or beneficial effects: By separating the analytical database from the relational database, the high-throughput batch write load and the low-latency random read load can be physically isolated, thereby eliminating the competition between the two loads for CPU, memory, and I / O, so that the ETL task does not affect the query service and ensures the stability of the overall system performance.

[0037] In some embodiments, the step of aggregating and calculating metrics on the business data using an analytical database based on business rules to obtain a data model includes the following steps: The business data is divided into themes according to the business rules to obtain business themes; The business data is initially aggregated based on the business theme to obtain theme data. Based on the business rules, the business objects are determined, and cross-topic data extraction and aggregation are performed on the topic data based on the business objects to obtain the indicator summary data. The data model is obtained by performing multi-dimensional correlation and aggregation processing on the summarized data of the indicators.

[0038] In this embodiment, periodic, computationally intensive batch data processing tasks are first executed through the data acquisition and computation layer. Raw data is extracted from various upstream business databases, and a complete ETL process, including data cleaning, multi-table joins, complex aggregation, and indicator calculation, is performed in the ArgoDB environment, ultimately generating a unified data model oriented towards the analytical theme. Taking transaction data processing as an example, after settlement each afternoon, transaction, settlement, and other business system data are batch-extracted to an analytical database for unprocessed storage. Then, based on business rules, the business data undergoes data aggregation and indicator calculation. Specifically, this embodiment, considering the characteristics of massive transaction data in the financial industry, constructs a theme-oriented preliminary aggregation system that integrates financial business logic. This lightly aggregates fragmented data scattered across multiple systems according to financial business rules, forming a directly usable unified thematic data view.

[0039] In some embodiments, the initial aggregation processing of the business data according to the business theme to obtain theme data includes the following steps: Based on the business theme, determine the business primary key and relationships of the data tables to obtain the associated data tables; The related data tables are merged vertically to obtain the topic data.

[0040] In this embodiment, business data is first divided into themes based on business rules, logically categorized into multiple themes such as transactions, settlements, and products. This can be achieved using a hierarchical namespace and prefix identifier specification, implementing this logical division at the physical table level. The primary keys of the data tables and the relationships between tables are determined based on the business themes. Thematic data is obtained by vertically merging the related data tables. Taking the transaction theme as an example, complete transaction records in a financial transaction system are often scattered across intraday and after-hours transaction tables. The intraday transaction table records every real-time transaction during the trading session, while the after-hours transaction table is generated after the settlement system closes. Specifically, by combining the characteristics of the financial industry, the relationship between the two transaction tables can be identified. The association key in the tables can be identified as the trading day to perform association analysis on the two transaction tables. Preliminary summarization yields the related data table. Through this summarization process, the transaction data originally scattered in two places is integrated into a complete transaction table, which includes both real-time intraday details and after-hours transaction data. Upper-layer applications can directly retrieve data from this thematic table without having to manually associate the two original tables, greatly simplifying subsequent processing logic and improving computational efficiency. Similar processing is applied to product themes, aggregating contract information scattered across futures and options contract tables. The futures contract table contains basic information such as contract code, contract month, delivery date, and contract multiplier, while the options contract table contains option-specific information such as strike price, option type, and expiration date. This invention vertically merges the two types of contracts by preserving their common characteristics and differences, generating a complete contract table. The "Contract Type" field distinguishes between futures and options, retaining the original values ​​for option-specific fields such as strike price, while leaving futures empty. The aggregated complete contract table unifies futures and options contracts under a single view, allowing upper-layer applications to directly access all contract information without distinguishing the original data source, greatly improving data access convenience and subsequent calculation efficiency. Through this cross-source data fusion and lightweight business aggregation, this embodiment achieves the effect of one-time fusion and multiple reuses. Upper-layer applications can directly retrieve data from the corresponding theme without repeatedly performing complex cross-source association operations, significantly improving overall computational efficiency.

[0041] In some embodiments, the step of performing cross-topic data extraction and aggregation on the topic data based on the business object to obtain summary indicator data includes the following steps: Based on the business object, the indicator data for different themes in the theme data is extracted to obtain the indicator data; The index data is aggregated and summarized according to different periods and different granularities to obtain the index summary data.

[0042] In this embodiment, a common indicator system is built around the core business object, and the data granularity is increased from fine to coarse. This allows for the accumulation and processing of frequently used and common information during the application process, thereby improving subsequent data processing and access performance. Taking "customer" as the core business object as an example, the system needs to extract indicator data from multiple thematic data and perform summary calculations. In the summary calculation stage, the system aggregates the extracted indicator data at different periods and granularities to generate common summary indicators for various analysis scenarios. For example, the daily transaction records of customers are summarized daily to generate a daily summary table of customer transactions; the daily summary is further accumulated monthly to generate a monthly summary table of customer transactions; similarly, a daily summary table of customer holdings is generated for holding data, and then a monthly summary table of customer holdings is generated monthly. This multi-granularity aggregation process allows for the construction of a complete public indicator system covering multiple time dimensions, including daily, weekly, monthly, quarterly, and yearly metrics. When performing analyses across different time spans, upper-level applications can directly retrieve data from the corresponding granularity summary tables, eliminating the need for repeated scanning and aggregation calculations of the original detailed data each time. This significantly improves query performance and user experience. This multi-granularity indicator accumulation around core business objects enables the system to complete most of the time-consuming aggregation calculations during the data processing phase, solidifying the calculation results in the form of public indicators and providing an efficient and consistent data foundation for upper-level data consumption.

[0043] After obtaining the aggregated indicator data, the system further performs multi-dimensional correlation and aggregation processing to construct a data model. This embodiment employs a dimensional modeling methodology, deeply correlated and calculated the data processed by the aggregation layer according to various dimensions required for business analysis, constructing a wide table that can directly serve the front-end application. Taking customer analysis as an example, the system correlates and aggregates multiple scattered summary tables generated by the aggregation layer, such as daily customer transaction summaries, daily customer position summaries, and daily customer fund summaries, according to time, customer, and product dimensions, generating a unified wide customer table containing indicators such as transaction volume, position, and fund for customers across various dimensions such as product type and date. In this wide table, each row represents the comprehensive status of a customer for a specific product on a specific trading day, covering dozens of indicator fields such as the customer's trading volume, position size, and margin usage on that day. This design, which integrates multiple dimensions of indicators into a single table, allows the front-end management dashboard to obtain all the required data by simply scanning a single wide table when performing various queries such as customer position structure analysis and customer transaction structure analysis, thus improving data processing efficiency.

[0044] In some embodiments, synchronizing the data model to a relational database using a data synchronization tool and generating an initial data view in the application interface based on the relational database includes the following steps: The data synchronization tool initiates a loading task to fully synchronize the data model to a relational database; the relational database stores the historical data of the data model. Obtain the front-end configuration of the application interface, and initiate a data query request to the relational database based on the front-end configuration; The data query request is processed by the relational database to obtain the query data and generate the corresponding initial data view.

[0045] In this embodiment, after the data acquisition and calculation layer completes its daily processing, the synchronization layer initiates a data synchronization task to synchronize the data model generated by the calculation layer to the data model of the query service layer. Because the data model undergoes highly aggregated processing, the overall data volume is small, allowing all historical data to be fully synchronized to the relational database while maintaining synchronization time within seconds. In the application interface, the management dashboard frontend initializes data upon user access, initiating initial data query requests for each chart component based on default configurations. These requests are sent to the relational database for execution and result return, which the frontend then uses to render the initial chart view. For example... Figure 3 As shown, the management dashboard supports data queries for four business scenarios: market overview, market analysis, transaction dynamics, and feature evaluation. Users can switch to a specific business scenario by clicking the button in the upper right corner of the page. Each business scenario consists of a set of independent sub-charts, which can be bar charts, pie charts, or line charts, etc. Taking the market analysis scenario as an example, clicking the market analysis button switches to the specific page, which consists of a set of charts, each independent, and the query conditions can be adjusted using the filter boxes within the charts. Figure 4 As shown, this sub-chart displays the position structure of various instruments in a market analysis scenario. It primarily serves to statistically analyze the position percentage of different client types on a specific trading day and for a particular instrument, presenting the data in a pie chart format. Above this sub-chart are filter boxes for both instrument and date. Clicking the filter boxes allows you to adjust the instrument and date filter criteria, and the chart will update immediately.

[0046] In some embodiments, generating a query statement in response to an interactive instruction on the initial data view to query the relational database and obtain query results includes the following steps: The interaction instructions are captured through the front-end framework of the application interface; the interaction instructions are used to adjust the filtering conditions of the initial data view. Based on the interactive instructions, the filtering conditions are determined, and the statement is constructed to obtain the query statement. The relational database is queried according to the query statement to obtain the query result.

[0047] In the application interface of this application embodiment, different business views such as market overview, market analysis, and transaction dynamics are presented as independent chart components. Each component is bound to an independent filter control, supporting users to perform multi-dimensional and freely combined data filtering. The front-end architecture adopts a responsive design. When a user operates the filter, specific interaction commands are captured. These commands are used to adjust the filtering conditions of the initial data view, thereby generating parameterized query statements independently for each affected chart. The query statements are sent concurrently to the back-end API gateway via asynchronous requests, and the back-end gateway routes these requests to a relational database. After the query results are returned, the front-end uses the chart library's data update interface to dynamically and partially refresh the charts, avoiding a full page reload and providing a smooth interactive experience.

[0048] In some embodiments, the step of determining the filtering conditions based on the interactive instructions and constructing the query statement includes the following steps: The interaction instruction is subjected to intent recognition to obtain the interaction intent, and the interaction intent is converted into the filtering conditions; The query statement is obtained by matching the interaction intent and passing the filtering conditions as parameters to the statement template for concatenation.

[0049] In this embodiment, the front-end framework monitors user actions on any chart filter, capturing interaction commands. For example, when a user adjusts the filter conditions of any chart, such as changing the time range or selecting a specific variety, the front-end framework captures this interaction event. The interaction intent is obtained through intent recognition. This intent can be preprocessed into a machine-understandable feature sequence. An intent classification model encodes and infers this feature sequence, calculating its matching degree with predefined intent categories to obtain the interaction intent. This interaction intent is then transformed—through semantic recognition and parameter extraction—to obtain the filter conditions. Matching the interaction intent yields a pre-constructed query template. The filter conditions are then passed as parameters to the matching template for concatenation, resulting in the final query statement. The construction of these query statements fully utilizes the underlying wide table structure; all filter conditions are converted into query clauses on a single table, avoiding complex subqueries or join operations.

[0050] The solutions of this application embodiment will be described in detail and explained below with reference to specific application examples: The embodiments of this application are applied to the technical fields of data processing, big data processing architecture optimization and enterprise-level data visualization. By constructing a layered decoupled and collaboratively optimized hybrid architecture system, it is suitable for analysis scenarios in the financial industry that require processing massive transaction data.

[0051] like Figure 5 As shown, the system consists of four logical layers, forming a complete data processing pipeline from data source to visualization. The data acquisition and computation layer is deployed on a big data cloud platform, with its core component being the ArgoDB distributed analytical database. This layer is responsible for executing periodic, computationally intensive batch data processing tasks. Specifically, it extracts raw data from various upstream business databases and performs a complete ETL process in the ArgoDB environment, including data cleaning, multi-table joins, complex aggregations, and indicator calculations, ultimately generating a unified data model oriented towards the analytical topic. This model adopts a wide table structure, integrating key dimensions and indicators from different business domains such as market, market data, and trading into a single logical table, aiming to eliminate table join overhead in subsequent queries.

[0052] The data synchronization layer, serving as the core pipeline connecting the computation layer and the query service layer, has its core component being the Transporter data synchronization tool. After the data acquisition and computation layers complete their daily processing, the synchronization layer initiates a data synchronization task, synchronizing the data from the 14 complete wide tables generated by the computation layer to the KunDB database in the query service layer. Because the 14 unified analysis wide tables undergo highly aggregated processing, the overall data volume is small, allowing for the full synchronization of all historical data to the KunDB database while maintaining synchronization time within seconds.

[0053] The core of the query service layer is the KunDB database, which serves as the dedicated query engine for the entire system. Compared to the analytical database ArgoDB, KunDB, as a relational database, offers faster query speeds and supports higher concurrency. Specifically, the KunDB database in the query service layer stores all historical data for 14 wide tables. For the front-end management dashboard application, historical data can be traced using date filters. All queries on the front-end charts can be converted into queries on specific wide tables.

[0054] The application presentation layer is the part that end users directly interact with, namely the management dashboard web application. This application frontend is built on a modern web framework and visualization libraries such as ECharts. Its key technology lies in efficiently translating user interaction intents into query requests to the underlying layers. In the application interface, different business views such as market overview, market analysis, and trading dynamics are presented as independent chart components. Each component is bound to an independent filter control, supporting users to perform multi-dimensional and freely combined data filtering. The frontend architecture adopts a responsive design. When a user interacts with a filter, a parameterized query statement is generated independently for each affected chart and sent concurrently to the backend API gateway via asynchronous requests. The backend gateway then routes these requests to the KunDB database. After the query results are returned, the frontend uses the chart library's data update interface to dynamically and partially refresh the charts, avoiding a full page reload and thus achieving a smooth interactive experience.

[0055] Please see Figure 6This application embodiment synchronizes data to the ArgoDB database in batches at scheduled times. At predetermined points, such as after daily settlement, the system initiates an ETL task at the data acquisition and computation layer, connecting to upstream business data sources to synchronize data to the ArgoDB database in batches. This data is loaded into the ArgoDB environment, executing a series of predefined data transformation rules and business computation logic. The computation process includes data filtering, data association, indicator calculation based on business rules, and finally, aggregation into multiple wide tables according to the analysis theme. The wide tables aggregate customer, market, and transaction data from different dimensions into a single table, avoiding the overhead of subsequent table associations. Full synchronization to KunDB is achieved using the Transporter synchronization tool: after the ETL task successfully completes and generates 14 unified analysis wide tables, the data synchronization process begins. Once the system detects the completion of the wide table generation task, it initiates a new loading task, which efficiently and completely synchronizes the wide table data to the KunDB database. The synchronized wide tables are full, containing all historical data, supporting historical data traceability. At the application presentation layer, the management dashboard frontend initializes data upon user access. The frontend initiates initial data query requests for each chart component based on the default configuration. These requests are sent to the query service layer, executed by KunDB, and the results are returned. The frontend then renders the initial chart view based on these results. When a user adjusts the filter conditions of any chart, such as changing the time range or selecting a specific product, the frontend framework captures this interaction event. The event handling logic independently constructs a parameterized SQL query statement for each chart whose conditions have changed, based on the current state of all its filters. The construction of these query statements fully utilizes the structure of the underlying wide table; all filter conditions are converted into query clauses on a single table, avoiding complex subqueries or join operations. The frontend sends these independent query requests to the backend service in parallel via asynchronous calls. Upon receiving the concurrent query requests, the backend service routes them to the KunDB database. KunDB's query optimizer and execution engine use pre-built B-Tree indexes to query and return the results.

[0056] Please see Figure 7 This application also provides a data processing apparatus based on a hybrid architecture system, which can implement the above-described data processing method based on a hybrid architecture system. The apparatus includes: The data acquisition and calculation module 701 is used to extract business data from the upstream business database according to a preset time point, and to perform data aggregation and indicator calculation processing on the business data based on business rules through an analytical database to obtain a data model. The data synchronization module 702 is used to synchronize the data model to a relational database through a data synchronization tool, and generate an initial data view on the application interface based on the relational database. The query service module 703 is used to generate a query statement in response to the interactive command of the initial data view to perform query processing on the relational database and obtain query results; The application display module 704 is used to render and update the initial data view based on the query results to obtain the target data view.

[0057] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0058] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0059] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0060] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0061] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0062] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0063] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0064] This application provides a data processing method and related equipment based on a hybrid architecture. The solution extracts business data from an upstream business database at a preset time point. An analytical database then aggregates and calculates metrics based on business rules to obtain a data model. This allows for data aggregation based on business rules, reducing the overhead of subsequent data association during queries and improving data processing efficiency. Furthermore, the solution synchronizes the data model to a relational database using a data synchronization tool and generates an initial data view on the application interface based on the relational database, enabling efficient and complete data synchronization. Additionally, the solution generates query statements in response to interactive commands on the initial data view, processes the query results in the relational database, and renders and updates the initial data view to obtain the target data view. This physically isolates high-throughput batch write loads from low-latency random read loads, ensuring overall system performance stability and improving data processing efficiency.

[0065] This application embodiment, through collaborative design between layers, separates ArgoDB and KunDB, physically isolating high-throughput batch write loads from low-latency random read loads. This fundamentally eliminates the competition for CPU, memory, and I / O between the two loads, ensuring that ETL tasks do not affect query services and guaranteeing the stability of the overall system performance. Simultaneously, by forming a unified wide table at the computation layer based on the cockpit's business needs, it lays the foundation for subsequent single-table queries. This application embodiment also provides that filtering any chart at the front end can be transformed into a query on a single wide table at the database level, reducing the performance overhead of online join calculations.

[0066] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0067] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0068] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0070] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0071] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0073] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A data processing method based on a hybrid architecture collaboration, characterized in that, The method includes the following steps: Business data is extracted from the upstream business database according to a preset time point. The business data is then aggregated and indexed by an analytical database based on business rules to obtain a data model. The data model is synchronized to a relational database using a data synchronization tool, and an initial data view is generated in the application interface based on the relational database. In response to the interactive command on the initial data view, a query statement is generated to perform query processing on the relational database and obtain query results; The initial data view is rendered and updated based on the query results to obtain the target data view.

2. The method according to claim 1, characterized in that, The process of aggregating and calculating metrics on business data using an analytical database based on business rules to obtain a data model includes the following steps: The business data is divided into themes according to the business rules to obtain business themes; The business data is initially aggregated based on the business theme to obtain theme data. Based on the business rules, the business objects are determined, and cross-topic data extraction and aggregation are performed on the topic data based on the business objects to obtain the indicator summary data. The data model is obtained by performing multi-dimensional correlation and aggregation processing on the summarized data of the indicators.

3. The method according to claim 2, characterized in that, The initial aggregation process of the business data based on the business theme to obtain theme data includes the following steps: Based on the business theme, determine the business primary key and relationships of the data tables to obtain the associated data tables; The related data tables are merged vertically to obtain the topic data.

4. The method according to claim 2, characterized in that, The process of extracting and aggregating cross-topic data based on the business object to obtain summary indicator data includes the following steps: Based on the business object, the indicator data for different themes in the theme data is extracted to obtain the indicator data; The index data is aggregated and summarized according to different periods and different granularities to obtain the index summary data.

5. The method according to claim 1, characterized in that, The process of synchronizing the data model to a relational database using a data synchronization tool and generating an initial data view on the application interface based on the relational database includes the following steps: The data synchronization tool initiates a loading task to fully synchronize the data model to a relational database; the relational database stores the historical data of the data model. Obtain the front-end configuration of the application interface, and initiate a data query request to the relational database based on the front-end configuration; The data query request is processed by the relational database to obtain the query data and generate the corresponding initial data view.

6. The method according to claim 1, characterized in that, The process of generating a query statement in response to an interactive instruction on the initial data view to query the relational database and obtain query results includes the following steps: The interaction instructions are captured through the front-end framework of the application interface; the interaction instructions are used to adjust the filtering conditions of the initial data view. Based on the interactive instructions, the filtering conditions are determined, and the statement is constructed to obtain the query statement. The relational database is queried according to the query statement to obtain the query result.

7. The method according to claim 6, characterized in that, The process of determining the filtering conditions based on the interactive instructions and constructing the query statement includes the following steps: The interaction instruction is subjected to intent recognition to obtain the interaction intent, and the interaction intent is converted into the filtering conditions; The query statement is obtained by matching the interaction intent and passing the filtering conditions as parameters to the statement template for concatenation.

8. A data processing device based on a hybrid architecture collaboration, characterized in that, The device includes: The data acquisition and calculation module is used to extract business data from the upstream business database according to a preset time point, and to perform data aggregation and indicator calculation processing on the business data based on business rules through an analytical database to obtain a data model. The data synchronization module is used to synchronize the data model to a relational database through a data synchronization tool, and generate an initial data view on the application interface based on the relational database. The query service module is used to generate a query statement in response to the interactive command of the initial data view to perform query processing on the relational database and obtain query results; The application display module is used to render and update the initial data view based on the query results to obtain the target data view.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images