Data processing method and device based on BS architecture, equipment and storage medium

Abstract

The invention discloses a data processing method, device and equipment based on a BS architecture and a storage medium, and the method comprises the steps: firstly obtaining a data analysis request sent by a front end, then analyzing the data analysis request, verifying the data analysis request with metadata information, and finally sending a data analysis request message passing the verification to a parallel processing database for processing. The technical problems that due to the fact that various self-service analysis tools exist in the market at present, the experience is low and the use threshold is high due to multiple functions, a server side and a client side need to be installed on the tools, configuration is complex, and the performance is low when the data size is huge are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, is applicable to the financial field, and in particular, to a data processing method, device, equipment and storage medium based on a BS architecture. Background Art

[0002] Data-driven modern large enterprises (such as banks) generally have a series of problems such as a large number of data reports, poor correlation, inconsistent caliber among reports, long development and launch time, and poor report coverage. Traditional reporting methods can hardly meet the requirements of enterprise data management.

[0003] To solve this problem, the industry has launched self-service analysis software that supports users to explore data sources and conduct self-service analysis based on different data needs. After exploring the required needs, they can be saved as templates, which can be easily called at any time to solve the problem of insufficient report flexibility.

[0004] There are a variety of self-service analysis tools on the market, such as Tableau, B+, and Fanruan. These self-service analysis tools have relatively complete functions, but they generally have the following problems:

[0005] 1. High threshold for use. Existing self-service analysis tools have many functions and relatively complex interfaces. Users need to be familiar with self-service analysis tools and existing data to use them freely, which is not friendly enough for ordinary business personnel;

[0006] 2. The tools are heavy. For example, the self-service analysis software Tableau needs to be installed on the server and client before it can be used. The web version of Tableau also needs to be installed on the server and perform complex settings before it can be used online.

[0007] 3. Performance needs to be improved. When performing self-service analysis on data sources with a volume of more than 10 million, freezes or timeout errors often occur.

[0008] Currently, there are a variety of self-service analysis tools on the market, which have high usage barriers, complex tool settings and poor performance. Technical problems need to be solved urgently. Summary of the invention

[0009] The present application provides a data processing method, apparatus, device and storage medium based on BS architecture, which first configures the basic library table of the data source and creates the corresponding data analysis model, then parses the data analysis request message sent by the front end, and verifies it with the metadata information in the data analysis model, and finally sends the verified data analysis request message to the parallel processing database for processing, thereby solving the technical problems that there are various self-service analysis tools on the market that have low user experience due to the large number of functions, and the tools need to install the server and client, resulting in complex configuration and poor performance when the amount of data is huge.

[0010] In view of this, the first aspect of the present application provides a data processing method based on a BS architecture, the method comprising:

[0011] Configure the data source basic library table and create the corresponding data analysis model;

[0012] Parse the data analysis request message sent by the front end and verify it with the metadata information in the data analysis model;

[0013] The data analysis request message that passes the verification is sent to the parallel processing database for processing.

[0014] Optionally, the steps of configuring a data source basic table and creating a corresponding data analysis model include:

[0015] Configure the data source acquisition channel and the corresponding SQL database model, and configure the data source basic library table;

[0016] The SQL database model is associated with the data source basic library table to create a corresponding data analysis model.

[0017] Optionally, the step of parsing the data analysis request message sent by the front end and verifying it with the metadata information in the data analysis model specifically includes:

[0018] Obtaining a data analysis request message sent by the front end through the data source acquisition channel;

[0019] Performing structural analysis and protocol field parsing on the message of the data analysis request;

[0020] The parsed data analysis request message is verified with the metadata information in the data analysis model.

[0021] Optionally, the step of verifying the parsed data analysis request message and metadata information specifically includes:

[0022] Performing data consistency check on the parsed data analysis request message and metadata information;

[0023] The parsed data analysis request message and metadata information are subjected to data permission verification.

[0024] Optionally, verifying the parsed data analysis request message and metadata information further includes:

[0025] Performing data integrity check on the parsed data analysis request message and metadata information;

[0026] The parsed data analysis request message and metadata information are verified for data timestamp.

[0027] Optionally, sending the verified data analysis request message to a parallel processing database for processing specifically includes:

[0028] Submitting the verified data analysis request message to the SQL database model, and converting the request message SQL statement corresponding to the data analysis request message;

[0029] The request message SQL statement is sent to the parallel processing database for processing.

[0030] Optionally, before parsing the data analysis request message sent by the front end and verifying it with the metadata information in the data analysis model, the following steps are also included:

[0031] Integrate analytical data from the data mart and the real-time data lake into the parallel processing database;

[0032] Calculate the frequency of field combination usage according to the analysis data, generate the field combination with the frequency of the preset position sorted, and send it to the front end, so that the data analysis request message includes the field combination. Send the data analysis request message that has passed the verification to the parallel processing database for processing. Send the data analysis request message that has passed the verification to the parallel processing database for processing.

[0033] A second aspect of the present application provides a data processing device based on a BS architecture, the device comprising:

[0034] The data analysis request unit is used to configure the basic database table of the data source and create the corresponding data analysis model;

[0035] An analysis request verification unit, used to parse the data analysis request message sent by the front end and verify it with the metadata information in the data analysis model;

[0036] The parallel processing unit is used to send the data analysis request message after verification to the parallel processing database for processing.

[0037] Optionally, the data analysis request unit specifically includes:

[0038] The configuration subunit is used to configure the data source acquisition channel and the corresponding SQL database model, and configure the data source basic library table;

[0039] The data analysis model subunit is used to associate the SQL database model with the data source basic library table to create a corresponding data analysis model.

[0040] Optionally, the analysis request verification unit specifically includes:

[0041] A message acquisition subunit, used to acquire a data analysis request message sent by the front end through the data source acquisition channel;

[0042] A message parsing subunit, used for performing structural analysis and protocol field analysis on the data analysis request message;

[0043] The message verification subunit is used to verify the parsed data analysis request message and the metadata information in the data analysis model.

[0044] Optionally, the message verification subunit is specifically used to perform data consistency verification on the parsed data analysis request message and metadata information, and to perform data permission verification on the parsed data analysis request message and metadata information.

[0045] Optionally, the message verification subunit is further used to perform data integrity verification on the parsed data analysis request message and metadata information, and to perform data timestamp verification on the parsed data analysis request message and metadata information.

[0046] Optionally, the parallel processing unit specifically includes:

[0047] An SQL statement conversion subunit is used to submit the verified data analysis request message to the SQL database model to perform request message SQL statement conversion corresponding to the data analysis request message;

[0048] The parallel processing subunit is used to send the request message SQL statement to the parallel processing database for processing.

[0049] Optionally, before obtaining the data analysis request sent by the front end, the following steps are also included:

[0050] A data integration unit, used to integrate analytical data from the data mart and the real-time data lake into the parallel processing database;

[0051] The analysis data calculation unit is used to calculate the frequency of use of field combinations according to the analysis data, generate the field combinations with the top 10 frequencies and send them to the front end, so that the data analysis request message includes the field combination.

[0052] A third aspect of the present application provides a data processing device based on a BS architecture, the device comprising a processor and a memory:

[0053] The memory is used to store the program code and transmit the program code to the processor;

[0054] The processor is used to execute the steps of the method for data processing based on BS architecture in the first aspect according to the instructions in the program code.

[0055] A fourth aspect of the present application provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the method of the first aspect above.

[0056] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0057] In the present application, a data processing method, apparatus, device and storage medium based on BS architecture are provided. First, a basic library table of a data source is configured, and a corresponding data analysis model is created. Then, a data analysis request message sent by a front end is parsed and verified with metadata information in the data analysis model. Finally, the data analysis request message after verification is sent to a parallel processing database for processing. By configuring a basic library table of a data source based on a BS architecture, and finally sending the data analysis request message after verification to a parallel processing database for processing, the technical problems that various self-service analysis tools currently available on the market have low user experience due to a large number of functions, and the tools need to install a server and a client, resulting in complex configuration and poor performance when the amount of data is huge are solved.

[0058] At the same time, in the present application, the frequency of use of field combinations is calculated based on the analysis data, and the field combinations with the top 10 frequencies are generated and sent to the front end, so that the data analysis request message includes the field combination. Common templates can be preset, so that the usage threshold for ordinary users can be lowered by pre-setting common templates. Users only need to click on the common template to analyze and download data online like using reports.

[0059] Furthermore, the verified data analysis request message is submitted to the SQL parser of the SQL database model for conversion into a request message SQL statement corresponding to the data analysis request message, so that users can perform self-service analysis of data through the web page anytime and anywhere without installing a client.

[0060] Furthermore, by sending the request message SQL statement to the parallel processing database for processing, data sources at the million, ten million, or billion level can be quickly processed and data analysis results can be presented, thus filling the tool gap in which ordinary users cannot process data at the million level or above through Excel.

[0061] At the same time, the data source acquisition channel and the corresponding SQL database model are configured, and the data source basic library table is configured, the SQL database model is associated with the data source basic library table, and the corresponding data analysis model is created. The entire process from configuring the data source, configuring self-service analysis, and data analysis is based on the BS architecture, without traditional clients, achieving beneficial effects such as lightweight, high performance, and easy maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A method flow chart of a data processing method based on a BS architecture in an embodiment of the present application;

[0063] Figure 2 This is a structural diagram of a data processing device based on a BS architecture in an embodiment of the present application;

[0064] Figure 3 It is a structural diagram of a data processing device based on a BS architecture in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. The non-company software tools or components that appear in the embodiments of the present application are merely examples and do not represent actual use.

[0066] The present application designs a data processing method based on the BS architecture, which first configures the basic library table of the data source and creates the corresponding data analysis model, then parses the data analysis request message sent by the front end, and verifies it with the metadata information in the data analysis model, and finally sends the verified data analysis request message to the parallel processing database for processing. By configuring the basic library table of the data source based on the BS architecture, and finally sending the verified data analysis request message to the parallel processing database for processing, it solves the technical problems that there are various self-service analysis tools on the market that have low experience due to the large number of functions, the tools need to install the server and the client, resulting in complex configuration and poor performance when the amount of data is huge.

[0067] For easier understanding, see Figure 1 , Figure 1 is a method flow chart of a data processing method based on a BS architecture in an embodiment of the present application, such as Figure 1 As shown, specifically:

[0068] S101. Configure the basic database table of the data source and create the corresponding data analysis model;

[0069] It is understandable that ordinary front-line users need a self-service analysis tool with a low threshold for use, a user-friendly interface, simple operation, integrated common functions, available online at any time, and able to process large amounts of data (above one million, users below one million tend to use Excel) to meet their daily work needs; for tool maintainers, self-service analysis tools must be stable and easy to maintain, and reducing maintenance manpower investment is a requirement for the tool.

[0070] Specifically, the step S101 may include:

[0071] Configure the data source acquisition channel and the corresponding SQL database model, and configure the data source basic library table;

[0072] The SQL database model is associated with the data source basic library table to create a corresponding data analysis model.

[0073] It is understandable that the self-service analysis tool, from configuring data sources, configuring self-service analysis, and data analysis, is based on the BS architecture. There is no traditional client. After the data source is uniformly configured, users can use it directly, which is simple and convenient to use. The process of configuring the base table is as follows:

[0074] (1)Configure data source

[0075] Added new data source to support mainstream databases within banks, linked via JDBC.

[0076] (2) Configure SQL assets (SQL database model)

[0077] ① Add sub-table: Generally, self-service analysis is used to analyze large and wide tables, detail SQL, or multiple table association SQL.

[0078] ②Configuration properties: mainly define the relevant properties of the fields involved in self-service analysis, such as dimensions, measures, data types, default display formats, etc., and support Excel batch import.

[0079] ③Configure controls: Configure the binding relationship between filter controls and default parameters and self-service analysis fields. By presetting the default display style of controls (tree, drop-down, calendar, interval, text box, etc.) and the method of obtaining candidate values ​​(json, sql), the loading efficiency of filter controls can be improved and the difficulty of user operation can be reduced.

[0080] ④Permission control: Prevent unauthorized access by configuring the corresponding menu URL.

[0081] (3) Configure associations (create corresponding data analysis models)

[0082] Supports configuration of associated reports, public templates, associated dimension tables, and associated analysis data analysis models. By configuring associations, users can easily access related reports / self-service analysis models. Configuring associated dimension tables can flexibly expand the dimensions / indicators of the current self-service analysis model.

[0083] S102, parsing the data analysis request message sent by the front end, and verifying it with the metadata information in the data analysis model;

[0084] It should be noted that the data analysis request message sent by the front end through the data source acquisition channel is obtained; the data analysis request message is structurally analyzed and the protocol fields are parsed; the parsed data analysis request message is verified with the metadata information in the data analysis model. It should be noted that before parsing the data analysis request message sent by the front end, the user needs to select analysis elements and screening conditions (or use preset templates) in the front-end analysis interface operation area, and then submit it to the back end for data clipping.

[0085] The process of data cutting and analysis by users through the front-end interface usually involves the following steps:

[0086] Select analysis elements: Users select the data elements they want to analyze on the front-end interface, which may be specific fields or variables in the dataset.

[0087] Set filter conditions: Users define filter conditions to narrow the scope of the data set. This may include selecting a specific date range, value range, text matching, or other logical conditions.

[0088] Apply preset templates: If the front end provides preset analysis templates, users can select a template to quickly apply a specific set of analysis elements and filtering conditions.

[0089] Submit the request: After the user completes their selections and settings, they submit their request to the backend server. This is usually done by clicking a "Submit" or "Analyze" button.

[0090] Backend processing: After receiving the data clipping request sent by the frontend, the backend server will process the data according to the user's selection and conditions. This may include operations such as data query, filtering, sorting, and aggregation.

[0091] Return results: After processing is completed, the backend sends the analysis results back to the frontend, and the frontend interface is displayed for users to view.

[0092] User Interaction: Users can view the results and potentially take further actions such as exporting the data, viewing detailed reports, or performing deeper analysis.

[0093] This process requires close collaboration between the front-end and back-end to ensure that user operations can be processed accurately and the results can be returned to users in a timely and effective manner.

[0094] When processing the data clipping request submitted by the front-end, the back-end server may:

[0095] Performance issues: If the amount of data is very large, the backend server may need to process a large amount of data, which may cause performance bottlenecks and affect response time.

[0096] Concurrent processing: When multiple users submit requests at the same time, the server needs to be able to efficiently handle concurrent requests to avoid service interruptions or delays.

[0097] Data security and privacy: It is very important to ensure the security and privacy of user data. The backend needs to implement appropriate security measures such as data encryption, access control, and audit logs.

[0098] Data consistency: Maintaining data consistency is a challenge in concurrent operations and distributed systems. It is necessary to ensure the accuracy and integrity of data during the editing and analysis process.

[0099] Error handling: The backend needs to be able to properly handle various error situations, such as invalid request parameters, non-existent data, system failures, etc., and provide clear error information to the frontend.

[0100] Resource management: The server needs to reasonably allocate and optimize resource usage, such as CPU, memory, and storage, to ensure stable operation of the system.

[0101] Scalability: As the number of users and data volume grows, the backend system needs to be able to flexibly expand to adapt to changing needs.

[0102] Compatibility issues: The backend needs to be able to handle requests from different front-end devices and browsers to ensure that all users have a consistent experience.

[0103] Data format and encoding issues: The data submitted by the front end may have different formats and encodings, and the back end needs to be able to parse and process this data correctly.

[0104] API design and documentation: The backend API needs to be clearly designed, easy to understand and use, and provide detailed documentation so that front-end developers can integrate and use it correctly.

[0105] Caching strategy: To improve performance, the backend may need to implement a caching strategy, but at the same time, it is also necessary to ensure that the cached data is updated in a timely manner and consistent.

[0106] Internationalization and localization: If the service is aimed at global users, the backend needs to support multiple languages ​​and localization to meet the needs of users in different regions.

[0107] Monitoring and logging: The backend needs to have monitoring and logging mechanisms to detect and resolve problems in a timely manner, as well as to perform performance analysis and optimization.

[0108] Comply with regulations and standards: When processing data on the back end, it is necessary to comply with relevant laws, regulations and industry standards, such as GDPR, HIPAA, etc.

[0109] It is understandable that before obtaining the data analysis request sent by the front end, it also includes:

[0110] Integrate analytical data from the data mart and the real-time data lake into the parallel processing database;

[0111] The frequency of use of the field combination is calculated based on the analysis data, and the field combinations with the top 10 frequencies are generated and sent to the front end, so that the data analysis request message includes the field combination.

[0112] Therefore, by analyzing the frequency of use of table field combinations, the fields that are frequently used in combination are filtered and combined into preset templates, which can optimize query efficiency and lower the user usage threshold.

[0113] Furthermore, it is understandable that the data in the data mart and real-time data lake data is integrated into the MPP high-performance database through the FlinkSQL batch-stream integration job, providing users with high-performance, quasi-real-time data analysis; preset self-service analysis templates. Administrators analyze the frequency of combined use of table fields, filter and combine the fields that are used in high-frequency combinations into preset templates, and optimize query efficiency, thereby lowering the user threshold while ensuring data analysis efficiency.

[0114] Apache Flink is an open source stream processing framework that supports high-throughput, low-latency data processing and can handle both batch and stream processing jobs. FlinkSQL is Flink's SQL interface, which allows users to use SQL statements to define data processing logic. FlinkSQL can be used to easily implement batch and stream integration jobs, integrating data from data marts and real-time data lakes into MPP high-performance databases.

[0115] The following are the general steps for data integration using FlinkSQL: Environment preparation: Make sure there is a running Flink cluster. Prepare the MPP database and obtain the JDBC driver. Define data sources: Use FlinkSQL to define data sources, which may include batch data sources (such as file systems, files in HDFS) and stream processing data sources (such as Kafka, Kinesis). Data transformation: Use FlinkSQL's transformation functions (such as SELECT, JOIN, GROUP BY, etc.) to process and transform data. Define data sinks: Use FlinkSQL to define data sinks, that is, the output target of the data, which here refers to the MPP database. Execute jobs: Submit the defined jobs to the Flink cluster for execution. Monitoring and management: Monitor the execution status of the job and manage the life cycle of the job.

[0116] As mentioned above, preset self-service analysis templates. Administrators can analyze the frequency of use of table field combinations and filter and combine fields that are used in high-frequency combinations into preset templates. This can be a common database optimization strategy to optimize query efficiency and lower the user usage threshold by analyzing the frequency of use of table field combinations. The following are steps to achieve this goal: Data collection: Collect database query logs and analyze user query patterns. Determine which field combinations are often used together in queries. Frequency analysis: Analyze the collected query data and count the frequency of occurrence of field combinations. This can be done using SQL queries or data analysis tools. Template design: Based on the analysis results, design preset templates and use fields that are used in high-frequency combinations as part of the template. The template can be a view, stored procedure, or predefined query statement. Index optimization: For high-frequency field combinations, consider creating composite indexes to improve query efficiency. Ensure that the creation of indexes does not negatively affect the write performance of the database. User interface simplification: In the user interface of the database management system, provide a query builder based on preset templates. Allow users to build queries by selecting field combinations instead of manually writing complete SQL statements. Documentation and training: Provide documentation and training materials to help users understand how to use preset templates. Educate users on how to use these templates to improve their work efficiency. Performance monitoring: Monitor the impact of preset templates on query performance. Regularly review and adjust templates to adapt to changes in user query patterns. Feedback loop: Establish a user feedback mechanism to collect user opinions and suggestions on preset templates. Continue to optimize templates and query performance based on user feedback.

[0117] The steps for setting up the aforementioned preset self-service analysis template are as follows: (1) Select required content such as data date; (2) Add analysis elements and filter according to the dimensions and fields required by the user; (3) Add filtering conditions to filter out unnecessary data; (4) Click Save as Template, name the template and save it. Subsequent clicks on the template will automatically bring in the corresponding fields and filtering conditions to improve usage efficiency; (5) Administrators are supported to set up public templates (that is, everyone can view the template), which lowers the user threshold and improves usage efficiency.

[0118] Further, performing a data consistency check on the parsed data analysis request message and metadata information;

[0119] Performing data permission verification on the parsed data analysis request message and metadata information;

[0120] Performing data integrity check on the parsed data analysis request message and metadata information;

[0121] The parsed data analysis request message and metadata information are verified for data timestamp.

[0122] In this embodiment, there are multiple ways to verify the parsed data analysis request message and the metadata information in the data analysis model. It can be that only one is satisfied, two are satisfied, or all are verified, and no specific limitation is made here.

[0123] After the back-end interface obtains the request, it will parse the message submitted by the user and verify the metadata information. Message parsing usually refers to the process of decoding and understanding the data packets (messages) in the data transmission process in a computer network. The message is a unit of data sent by the source device and transmitted to the target device through the network. Message parsing is crucial to ensure the correct transmission and reception of data. It involves the following key steps: Identify the message type: determine which protocol the message is a data packet, such as TCP, UDP, HTTP, FTP, etc. Message structure analysis: parse the message header and data payload according to the protocol specification. The message header usually contains information such as the source address, destination address, port number, protocol type, sequence number, etc., while the data payload contains the actual transmitted data. Error detection: Check whether the message has errors during transmission, such as loss, duplication, or damage. This is usually done through mechanisms such as checksum or CRC (cyclic redundancy check). Protocol field parsing: parse the various fields in the message header according to the protocol specification, such as sequence number, confirmation number, window size, etc. Data reassembly: When necessary, multiple message fragments are reassembled into complete data, which is particularly important in connection-oriented protocols such as TCP. Application layer processing: The parsed data is passed to the application for further processing or display.

[0124] In practical applications, message parsing may involve complex protocol stacks and multiple network protocols. For example, message parsing of an HTTP request may include parsing the request line (such as GET / index.html HTTP / 1.1), request header (such as Host: www.example.com), blank line, and request body.

[0125] Parsing a message means converting the original message data into a readable format, while metadata verification ensures that the parsed data matches the expected metadata information. The following is a detailed description of these two steps:

[0126] Parsing a message usually includes the following: Message header: Contains metadata about the message, such as sender, receiver, message size, timestamp, protocol type, etc. Payload: The actual data portion of the message, which may contain text, images, audio, or other types of data. Message trailer: May contain checksums, signatures, or other information used to verify the integrity of the message. When parsing a message, you may encounter a variety of error types that may affect the accuracy and integrity of the data. The following are some common error types,

[0127] Special attention should be paid to: Format error: The message may not conform to the expected format specification, such as incorrect field length, missing necessary separators, or containing illegal characters. Protocol mismatch: The message may use the wrong protocol or version, resulting in the parser being unable to correctly identify and process it. Data type error: The data in the message may not match the expected data type, such as numbers being incorrectly parsed as strings. Encoding problem: The message may use incorrect character encoding, resulting in garbled text data or incorrect display. Checksum error: The checksum or CRC of the message may not match, indicating that the data may have been damaged during transmission. Sequence error: In a message stream that needs to be processed in sequence, there may be lost or duplicate messages, resulting in discontinuous data sequences. Timestamp error: The timestamp of the message may be incorrect, resulting in problems with data sorting or synchronization. Metadata inconsistency: The metadata in the message header may be inconsistent with the message content or other metadata records. Permission and security issues: The message may not pass the appropriate security verification or contain unauthorized data access requests. Resource limitation error: Resource limitations may be encountered during the parsing process, such as insufficient memory or insufficient processing power, resulting in parsing failure. Parser error: The parser itself may have defects, resulting in the inability to parse the message correctly. Data overflow: The data in the message may exceed the maximum allowed value of the field, resulting in a parsing error. Protocol stack error: In a multi-layer protocol stack, errors in the lower-layer protocol may affect the message parsing of the upper-layer protocol. Network problem: Network delay, packet loss or jitter may cause message fragmentation or reassembly errors.

[0128] Metadata verification usually involves the following steps: Verify data integrity: Ensure that the message has not been tampered with or damaged during transmission. Check data consistency: Ensure that the data in the message matches the metadata record in the database or storage system. Verify data type: Ensure that the data type in the message is consistent with the type defined in the metadata. Permissions and access control: Check whether the sender and receiver have permission to exchange data. Timestamp and version control: Verify whether the timestamp of the message is within a reasonable time range and whether the message is the latest version.

[0129] In the actual verification process, you may need to perform the following operations: Comparison verification: Compare the parsed data with the metadata records in the database to check if there are any differences. Error handling: If data inconsistencies are found, it is necessary to determine whether it is a data error or a metadata error, and take appropriate corrective measures. Record logs: Record all problems found during the verification process to facilitate subsequent audits and problem tracking. Update metadata: If the parsed data is correct, but the metadata record is incorrect, you may need to update the metadata to reflect the actual data status.

[0130] Therefore, the specific verification steps are:

[0131] 1) Whether the analysis element is valid: Verify whether the analysis element selected by the user is consistent with the attributes in the self-service analysis configuration;

[0132] 2) Whether the filtering conditions are valid: Check whether the parameter name, format type, etc. of the filtering conditions are consistent with the configuration.

[0133] S103: Send the verified data analysis request message to the parallel processing database for processing.

[0134] It can be understood that the data analysis request message after verification is submitted to the SQL parser of the SQL database model to convert the request message SQL statement corresponding to the data analysis request message;

[0135] The request message SQL statement is sent to the parallel processing database for processing.

[0136] After verification, it is submitted to the SQL parser of the SQL database model, which assembles the analysis elements, function settings, filter conditions, etc. in the user's analysis template request into executable SQL. SQL is submitted to the high-performance MPP database cluster for execution through jdbc connection; the front end finally provides visual data analysis results in the form of tables and graphics.

[0137] The process of converting user requests into executable SQL statements involves the following steps:

[0138] Request parsing: First, the user's request needs to be parsed, which may include natural language processing or parsing of a specific query language. If it is a natural language request, it may be necessary to use natural language processing technology to convert it into a structured query. Syntax analysis: The parser breaks down the user's request into grammatical units, such as keywords, operators, identifiers, etc., and builds a syntax tree. This process ensures that the request complies with the grammatical rules of the SQL language. Semantic analysis: At this stage, the parser checks whether the parsed syntax tree is semantically meaningful. This includes checking whether the table name and column name exist, whether the data type matches, and whether the user has permission to perform the requested operation. Query optimization: Before generating executable SQL statements, there is usually a query optimizer to improve the performance of the query. The optimizer may rewrite the query, select a more effective index, or change the order of the connection operation. Generate an execution plan: The optimized query is converted into one or more execution plans, which describe in detail how to access and operate the data in the database. Execute SQL: The execution plan is sent to the database engine, and the database engine executes the SQL statement according to the plan to retrieve or modify data. Return results: After execution, the results will be returned to the user.

[0139] In this process, you may need to pay attention to the following aspects: Security: Ensure that the user's request will not cause SQL injection attacks or other security issues. Error handling: If the request cannot be converted into a valid SQL statement, a clear error message should be provided. Performance: The generated SQL statement should be as efficient as possible to reduce the database load and improve response time.

[0140] The aforementioned sending of the request message SQL statement to the parallel processing database for processing involves several key steps in the process of connecting to a high-performance MPP (Massively Parallel Processing) database cluster through JDBC and executing SQL statements. The following is a general process: Add JDBC driver: Make sure that the project contains the JDBC driver for the corresponding MPP database. This usually means adding the driver's JAR file to the class path of the project. Establish a database connection: Use the DriverManager provided by the JDBC driver to establish a connection with the MPP database cluster. The connection string usually contains the database address, port, database name, and other connection parameters. Create Statement or PreparedStatement: Create Statement or PreparedStatement objects through the connection object, which will be used to execute SQL statements. Execute SQL statements: Use executeQuery, executeUpdate, or execute methods to execute SQL statements. For query operations, executeQuery can be used and return a ResultSet object; for update operations such as INSERT, UPDATE, or DELETE, executeUpdate can be used. Processing results: For query operations, traverse the ResultSet object to process the query results. For update operations, the executeUpdate method returns an integer value indicating the number of rows affected. Close resources: After completing database operations, close ResultSet, Statement, and database connection to release resources. Exception handling: Capture and handle possible SQL exceptions to ensure the robustness of the program.

[0141] This embodiment can be applied to the BS architecture, which is the Browser / Server architecture, a software system architecture model mainly used to build Web-based applications. In this architecture, users use browsers as clients to access resources and services on the server. The BS architecture is in contrast to the CS (Client / Server) architecture, which usually requires the installation of special client software on the user's computer.

[0142] A data processing method based on the BS architecture is provided in this embodiment. The method first obtains the data analysis request sent by the front end, then parses the data analysis request and verifies it with the metadata information, and finally sends the verified data analysis request message to the parallel processing database for processing. By configuring the data source basic library table based on the BS architecture, the verified data analysis request message is finally sent to the parallel processing database for processing. This solves the technical problems of low experience due to the large number of functions, high usage threshold, complex configuration and poor performance when the data volume is huge caused by the presence of a variety of self-service analysis tools on the market.

[0143] At the same time, in this embodiment, a common template can be preset based on the analysis data to calculate the frequency of field combination use, so that the usage threshold for ordinary users is lowered by presetting the common template. Users only need to click on the common template to analyze and download data online like using a report.

[0144] Furthermore, the verified data analysis request message is submitted to the SQL parser of the SQL database model for conversion into a request message SQL statement corresponding to the data analysis request message, so that users can perform self-service analysis of data through the web page anytime and anywhere without installing a client.

[0145] Furthermore, by sending the request message SQL statement to the parallel processing database for processing, data sources at the million, ten million, or billion level can be quickly processed and data analysis results can be presented, thus filling the tool gap in which ordinary users cannot process data at the million level or above through Excel.

[0146] At the same time, the data source acquisition channel and the corresponding SQL database model are configured, and the data source basic library table is configured, the SQL database model is associated with the data source basic library table, and the corresponding data analysis model is created. The entire process from configuring the data source, configuring self-service analysis, and data analysis is based on the BS architecture, without traditional clients, achieving beneficial effects such as lightweight, high performance, and easy maintenance.

[0147] See also Figure 2 , Figure 2 Schematic diagram of the structure of the data processing device based on the BS architecture in the embodiment of the present application. Figure 2 As shown, specifically:

[0148] The data analysis request unit 201 is used to configure the data source basic library table and create the corresponding data analysis model;

[0149] An analysis request verification unit 202 is used to parse the data analysis request message sent by the front end and verify it with the metadata information in the data analysis model;

[0150] The parallel processing unit 203 is used to send the verified data analysis request message to the parallel processing database for processing.

[0151] In the present embodiment, a data processing device based on the BS architecture is provided, which obtains the data analysis request sent by the front end first, then parses the data analysis request and verifies it with the metadata information, and finally sends the verified data analysis request message to the parallel processing database for processing. By configuring the data source basic library table based on the BS architecture, the verified data analysis request message is finally sent to the parallel processing database for processing, thereby solving the technical problems of low experience due to the large number of functions, high usage threshold, complex configuration caused by the need to install the server and client, and poor performance when the data volume is huge due to the various self-service analysis tools currently on the market.

[0152] At the same time, in this embodiment, a common template can be preset based on the analysis data to calculate the frequency of field combination use, so that the usage threshold for ordinary users is lowered by presetting the common template. Users only need to click on the common template to analyze and download data online like using a report.

[0153] Furthermore, the verified data analysis request message is submitted to the SQL parser of the SQL database model for conversion into a request message SQL statement corresponding to the data analysis request message, so that users can perform self-service analysis of data through the web page anytime and anywhere without installing a client.

[0154] Furthermore, by sending the request message SQL statement to the parallel processing database for processing, data sources at the million, ten million, or billion level can be quickly processed and data analysis results can be presented, thus filling the tool gap in which ordinary users cannot process data at the million level or above through Excel.

[0155] At the same time, the data source acquisition channel and the corresponding SQL database model are configured, and the data source basic library table is configured, the SQL database model is associated with the data source basic library table, and the corresponding data analysis model is created. The entire process from configuring the data source, configuring self-service analysis, and data analysis is based on the BS architecture, without traditional clients, achieving beneficial effects such as lightweight, high performance, and easy maintenance.

[0156] The present application also provides another data processing device based on BS architecture, such as Figure 3 As shown, the device 10 includes:

[0157] One or more processors 110 and memory 120, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0158] The processor 110 is used to complete various control logics of the device 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0159] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for online processing of fixed asset data in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for online processing of fixed asset data in the above method embodiment.

[0160] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0162] Configure the data source basic library table and create the corresponding data analysis model;

[0163] Parse the data analysis request message sent by the front end and verify it with the metadata information in the data analysis model;

[0164] The data analysis request message that passes the verification is sent to the parallel processing database for processing.

[0165] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the above steps S101 to S103 are implemented.

[0166] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0167] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0168] In this embodiment, a data processing device based on the BS architecture is provided, which obtains the data analysis request sent by the front end, parses the data analysis request, verifies it with the metadata information, and finally sends the verified data analysis request message to the parallel processing database for processing. By configuring the data source basic library table based on the BS architecture, the verified data analysis request message is finally sent to the parallel processing database for processing. This solves the technical problems of low experience due to the large number of functions, high usage threshold, complex configuration and poor performance when the data volume is huge caused by the presence of a variety of self-service analysis tools on the market.

[0169] At the same time, in this embodiment, a common template can be preset based on the analysis data to calculate the frequency of field combination use, so that the usage threshold for ordinary users is lowered by presetting the common template. Users only need to click on the common template to analyze and download data online like using a report.

[0170] Furthermore, the verified data analysis request message is submitted to the SQL parser of the SQL database model for conversion into a request message SQL statement corresponding to the data analysis request message, so that users can perform self-service analysis of data through the web page anytime and anywhere without installing a client.

[0171] Furthermore, by sending the request message SQL statement to the parallel processing database for processing, data sources at the million, ten million, or billion level can be quickly processed and data analysis results can be presented, thus filling the tool gap in which ordinary users cannot process data at the million level or above through Excel.

[0172] At the same time, the data source acquisition channel and the corresponding SQL database model are configured, and the data source basic library table is configured, the SQL database model is associated with the data source basic library table, and the corresponding data analysis model is created. The entire process from configuring the data source, configuring self-service analysis, and data analysis is based on the BS architecture, without traditional clients, achieving beneficial effects such as lightweight, high performance, and easy maintenance.

[0173] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A data processing method based on BS architecture, characterized in that: include: Configure the data source basic library table and create the corresponding data analysis model; Parse the data analysis request message sent by the front end and verify it with the metadata information in the data analysis model; The data analysis request message that passes the verification is sent to the parallel processing database for processing.

2. The data processing method based on BS architecture according to claim 1, characterized in that: The steps to configure the data source basic table and create the corresponding data analysis model include: Configure the data source acquisition channel and the corresponding SQL database model, and configure the data source basic library table; The SQL database model is associated with the data source basic library table to create a corresponding data analysis model.

3. The data processing method based on BS architecture according to claim 1, characterized in that: The step of parsing the data analysis request message sent by the front end and verifying it with the metadata information in the data analysis model specifically includes: Obtaining a data analysis request message sent by the front end through the data source acquisition channel; Performing structural analysis and protocol field parsing on the data analysis request message; The parsed data analysis request message is verified with the metadata information in the data analysis model.

4. The data processing method based on BS architecture according to claim 3 is characterized in that: The step of verifying the parsed data analysis request message and metadata information specifically includes: Performing data consistency check on the parsed data analysis request message and metadata information; The parsed data analysis request message and metadata information are subjected to data permission verification.

5. The data processing method based on BS architecture according to claim 3 is characterized in that: Verifying the parsed data analysis request message and metadata information also includes: Performing data integrity check on the parsed data analysis request message and metadata information; The parsed data analysis request message and metadata information are verified for data timestamp.

6. The data processing method based on BS architecture according to claim 5, characterized in that: Sending the verified data analysis request message to the parallel processing database for processing specifically includes: Submitting the verified data analysis request message to the SQL database model, and converting the request message SQL statement corresponding to the data analysis request message; The request message SQL statement is sent to the parallel processing database for processing.

7. The data processing method based on BS architecture according to claim 6, characterized in that: Before parsing the data analysis request message sent by the front end and verifying it with the metadata information in the data analysis model, it also includes: Integrate analytical data from the data mart and the real-time data lake into the parallel processing database; The frequency of use of the field combination is calculated according to the analysis data, and the field combination with the preset frequency before sorting is generated and sent to the front end, so that the data analysis request message includes the field combination.

8. A data processing device based on BS architecture, characterized in that: include: The data analysis request unit is used to configure the basic database table of the data source and create the corresponding data analysis model; An analysis request verification unit, used to parse the data analysis request message sent by the front end and verify it with the metadata information in the data analysis model; The parallel processing unit is used to send the data analysis request message that has passed the verification to the parallel processing database for processing.

9. A data processing device based on BS architecture, characterized in that: The device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method based on the BS architecture according to any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the data processing method based on the BS architecture as described in any one of claims 1-7.

Images