Visual data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a visual data processing method and device, computer equipment and a storage medium. The method comprises the steps that financial data are acquired and loaded to a pre-generated data form, and the data form is generated based on a data display demand; creating an associated view based on the asset structure information between the financial data, and extracting data from the data form by adopting the associated view; processing the extracted data to obtain to-be-displayed data; and loading the to-be-displayed data to a target data warehouse for display. By adopting the method, massive financial data can be quickly converted into display data meeting requirements according to user requirements, and a display result with high processing speed and high accuracy is provided for a user.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and in particular to a visual data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of network technology, the speed of data generation and dissemination is accelerating, and the value of data will also decline rapidly. Obtaining valuable data from large data sets for display and user analysis has become an important basis for helping enterprises make business decisions.

[0003] Due to the differences between different operating systems and the different databases used by various software platforms or inconsistent database versions, problems such as data redundancy, missing data, and conflicts often occur during data processing, affecting the accuracy of data display results. Summary of the invention

[0004] Based on this, it is necessary to provide a visual data processing method, device, computer equipment and storage medium to address the above technical problems.

[0005] In a first aspect, the present application provides a method for processing visual data, the method comprising:

[0006] Obtain financial data and load it into a pre-generated data form, wherein the data form is generated based on data display requirements;

[0007] Creating an associated view based on the asset structure information between the financial data, and extracting data from the data form using the associated view;

[0008] Process the extracted data to obtain the data to be displayed;

[0009] The data to be displayed is loaded into the target data warehouse for display.

[0010] In one embodiment, the steps before obtaining the financial data include:

[0011] Acquire document information of a data system to be accessed, and perform interface encapsulation on the data system according to the document information;

[0012] Financial data is obtained from the data system after interface encapsulation.

[0013] In one embodiment, the step of generating a data form based on data display requirements includes:

[0014] Select multiple corresponding fields according to data display requirements;

[0015] The table structure is designed according to the mapping relationship between the fields to construct the data form.

[0016] In one embodiment, creating an associated view based on the asset structure information between the financial data, and extracting data from the data form using the associated view, includes:

[0017] Acquire a mapping relationship between the fields according to the asset structure information between the financial data, and determine a query logic based on the mapping relationship;

[0018] Write a query statement according to the mapping relationship and query logic;

[0019] Encapsulating the query statement as an associated view;

[0020] In response to a query instruction, query the data form through the associated view;

[0021] The query result is used as the data to be extracted.

[0022] In one embodiment, the processing of the data to be displayed includes: any one or more of data cleaning, data conversion and data integration.

[0023] In one of the embodiments, the processing of the extracted data further includes: encrypting the sensitive data during the data conversion process.

[0024] In one embodiment, the data to be displayed is loaded into a plurality of database tables of the target data warehouse; and displaying the data to be displayed includes:

[0025] Setting key fields, wherein the key fields include one or more of table name, column name, creation time and update time;

[0026] Query all database tables in the target data warehouse, extract key fields, and generate temporary tables based on the extracted key fields;

[0027] After receiving the query statement, the temporary table is queried, the corresponding database form is located according to the query result, and the data to be displayed in the database form is extracted for display.

[0028] In a second aspect, the present application also provides a visual data processing device. The device comprises:

[0029] An acquisition module, used to acquire financial data and load it into a pre-generated data form, wherein the data form is generated based on data display requirements;

[0030] A data extraction module, used to create an associated view based on the asset structure information between the financial data, and extract data from the data form using the associated view;

[0031] A data processing module is used to process the extracted data to obtain data to be displayed;

[0032] The display module is used to load the data to be displayed into the target data warehouse for display.

[0033] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method steps described in any one of the first aspects are implemented.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0035] The above-mentioned visual data processing method, device, computer equipment and storage medium have at least the following advantages:

[0036] This application loads financial data obtained from multiple databases into pre-generated data forms, and uses associated views to extract required data from the data forms, which improves query speed and efficiency while also increasing data security; finally, the extracted data is processed and loaded into the target data warehouse for users to review and display. This application can quickly convert massive amounts of financial data into display data that meets the requirements according to user needs, providing users with fast processing speed and high accuracy display results. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 An application environment diagram of a visual data processing method in an embodiment;

[0040] Figure 2 A flowchart of a method for visualizing data processing in one embodiment;

[0041] Figure 3A schematic diagram of a flow chart of an interface encapsulation step in an embodiment;

[0042] Figure 4 A schematic diagram of a flow chart of steps for generating a data form in one embodiment;

[0043] Figure 5 A schematic diagram of a flow chart of the steps of extracting data from a data form using an associated view in one embodiment;

[0044] Figure 6 is a structural block diagram of a visual data processing device in one embodiment;

[0045] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0047] While some exemplary embodiments of the present invention have been described for the purpose of illustration, it should be understood that the present invention may be implemented in other ways not specifically shown in the drawings.

[0048] The visual data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0049] Among them, the terminal 102 can send instructions to the server 104, so that the server 104 obtains financial data from multiple databases for processing. For example, the server 104 loads the obtained financial data into a pre-generated data form, wherein the data form is generated based on the data display requirements. Then, an associated view is created based on the asset structure information between the financial data, and the associated view is used to extract the data to be displayed from the data form. Finally, the extracted data is processed to obtain the data to be displayed, and the data to be displayed is loaded into the target data warehouse for display. In this way, the server 104 feeds back the display results to the terminal 102.

[0050] The above-mentioned visual data processing method loads the financial data obtained from multiple databases into a pre-generated data form, and uses the associated view to extract the required data from the data form, which improves the query speed and efficiency while also increasing the security of the data; finally, the extracted data is processed and loaded into the target data warehouse for users to view and display. This application can convert massive amounts of financial data into display data that meets the requirements according to user needs, providing users with display results with fast processing speed and high accuracy.

[0051] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0052] In an exemplary embodiment, the present application provides a method for processing visual data. Figure 1 The server 104 in FIG. 1 is used as an example for explanation.

[0053] See also Figure 2 , Figure 2 The following is a flow chart of a visual data processing method according to the present embodiment, which specifically includes the following steps:

[0054] Step S202, obtain financial data and load it into a pre-generated data form, wherein the data form is generated based on data display requirements.

[0055] Specifically, the financial data of an enterprise includes internal data and external data, wherein the internal data usually involves the assets, liabilities, income, expenses, profits, cash flows, financial ratios and equity of the enterprise; the external data usually involves bank data, market data, industry data and economic data. These financial data can be obtained in a variety of ways, such as the financial software used by the enterprise, the reconciliation platform provided by the bank, manual entry or intelligent OCR recognition of the original file. The above financial data is stored in public databases, as well as multiple databases within the enterprise and the bank for users to call. Exemplarily, in this embodiment, a voucher list is obtained through an ERP (Enterprise Resource Planning) system, and the voucher list is used as financial data.

[0056] Step S204: creating an associated view based on the asset structure information between the financial data, and extracting data from the data form using the associated view.

[0057] Specifically, financial data belongs to multiple sections, and the key information within each section is interrelated. Taking the asset and liability section as an example, it mainly involves asset and liability information, asset structure information for the current year, comparison information of major asset items, comparison information of major liability items, detailed information on capital structure, and detailed information on liability structure. The asset structure information represents the correlation between these information. By parsing the asset structure information, the large amount of messy financial data can be divided into multiple categories according to display needs, and then multiple data forms can be pre-built based on the categories, and the financial data of the same category can be loaded into the corresponding data form, so as to facilitate the subsequent rapid analysis of the data. For example, taking the service object as a bank customer as an example, the subject classification of data according to customer needs in this application can be roughly divided into:

[0058] Customer data: such as basic customer information, transaction information, credit records, etc.;

[0059] Product data: information about goods or services provided by the bank, such as loans, deposits, financial products, etc.;

[0060] Transaction data: records various transaction activities between banks and customers, including transfers, deposits, withdrawals, etc.;

[0061] Channel data: describes the channels through which customers interact with banks, such as online banking, ATMs, and bank counters;

[0062] Address data: including the bank’s physical address and electronic address;

[0063] Asset data: asset information of customers and banks, such as deposit balances, loan balances, etc.;

[0064] Financial data: including bank financial statements, budgets, account details, etc.;

[0065] Marketing data: data recording the bank’s marketing activities, such as advertising, promotional activities, etc.;

[0066] Organizational data: describes the bank’s own organizational structure, employee information, etc., and groups and summarizes them.

[0067] Based on the above subject classification, for example, the above asset and liability section, the final data forms include liability structure details table, capital structure details table, balance sheet, cash table, liability item comparison table, asset item comparison table, liability structure table, and asset structure table.

[0068] Furthermore, due to the large amount of financial data, the structure of the data form is relatively complex, and due to the correlation between the fields of each data form, there is often a situation where the data in one form corresponds to multiple other forms, resulting in a long query time. For this reason, complex query statements with many conditions and requiring multi-table joint query are created as associated views in advance, and the query results are placed in the associated view, so that the required data can be extracted from the associated view later, thereby improving the query speed and efficiency, and more finely controlling the user's access to the data, thereby increasing the security of the data. For example, the balance sheet takes the total assets, total liabilities, debt-to-asset ratio, current asset ratio, structure, company, and time fields to create a view; the asset structure table and the liability structure table take the structure and amount fields to create a view; the asset structure detail table and the liability structure detail table take the storage time, structure, and amount fields to create a view; the monetary funds table takes the storage time, indicators, and amount fields to create a view; the asset item comparison table and the liability item comparison table take the item and amount fields to create a view.

[0069] Step S206, processing the extracted data to obtain data to be displayed.

[0070] Specifically, the data in each data form is raw data, which may have problems such as data duplication, data missing, inconsistent data format, and low data security. It needs to be cleaned and format converted to data that meets the display requirements. In addition, in order to meet user needs, the data needs to be further integrated into a master table according to thematic classification, and stored for subsequent display.

[0071] Step S208: Load the data to be displayed into the target data warehouse for display.

[0072] Specifically, a data warehouse is a subject-oriented, integrated, non-volatile, and time-varying data set used to support the decision-making process of an enterprise or organization. It usually includes four layers, among which the ODS (Operation Data Store) layer is the original data layer, which is used to store and load the original data, and the data remains in its original form without processing; the DWD (Data warehouse detail) layer is the detail data layer, which is used to clean the ODS layer data; the DWS (datawarehouse service) layer is the service data layer, which is used to perform light aggregation based on DWD; the ADS (Application Data Store) layer is the application data layer, which is used to provide data for various statistical reports. The data extracted from the data form is stored in the ODS layer, which provides a basis for subsequent data processing; the processed data to be displayed is stored in the ADS layer.

[0073] Furthermore, to ensure the real-time nature of the data, it is necessary to create triggers in the target data warehouse, which are triggered when the data changes. The triggers can track data changes and automatically cascade modifications to tables in the database to ensure that when the data in one table is modified, related tables can also be updated accordingly, thereby maintaining data consistency and ensuring the real-time nature of data in the ADS layer.

[0074] The above-mentioned visual data processing method generates multiple data forms in advance according to the data display requirements, divides the huge financial data into multiple different topics and loads them into corresponding data forms; at the same time, the complex query statements in the data form with many conditions and requiring multi-table joint query are created as associated views, so that the required data can be extracted from the associated views later, thereby improving the query speed and efficiency, and more finely controlling the user's access to the data, increasing the security of the data, and ultimately providing users with fast processing speed and high accuracy display results.

[0075] See also Figure 3 Optionally, the steps before obtaining financial data include:

[0076] Step S302, obtaining document information of the data system to be accessed, and performing interface encapsulation on the data system according to the document information.

[0077] Step S304, obtaining financial data from the data system after interface encapsulation.

[0078] Specifically, in order to simplify the communication process between the client and the server, the client only needs to call the encapsulated interface without having to worry about the underlying processing details, thereby improving development efficiency and reducing duplicate code. This application encapsulates the interface of the accessed data system. Taking the ERP system as an example, the data system to be accessed uses the http interface. The document information includes the functions, parameters, and request methods of the interface. By defining the interface request method, configuring request parameters, sending HTTP requests, processing HTTP responses, using generics and deserialization, logging, and monitoring, the above logic can be encapsulated into one or more tool classes to provide a concise interface for the business code to call.

[0079] Furthermore, a scheduled task can be set to obtain data from the data system at a fixed time every day. For example, it can be set to start at 0:00 every day, and the following operations are performed in sequence: obtain the domain name of the data center where the tenant is located, calculate the encryption signature according to the current timestamp, obtain the dynamic password (access_token), record the number of records in the current database, clear the database, access the data interface in a loop to obtain paging data, data entry, record the number of records in the updated database and output the log. If the above steps are successfully executed, wait for the same time the next day to execute again, otherwise repeat the above steps. If the execution fails three times, an error is reported. For example, the time intervals of the three times are 10 minutes, 50 minutes, and 250 minutes respectively.

[0080] Considering that the data in the ERP system will be updated, the data acquisition method of this application is full acquisition. At the same time, the acquired data will be backed up regularly, and there is a failure retry mechanism, so the transaction is not opened in this embodiment to reduce the burden on the server.

[0081] Optionally, the acquired financial data is signature verified, wherein the ERP system uses an encryption algorithm and a key to calculate the request parameters to obtain a signature; the generated signature value is attached to the data packet and output to the user end, and the user performs signature verification after receiving the data packet to ensure the security and integrity of the data. In this embodiment, the encryption algorithm uses HmacSHA256, and the key is usually divided into a public key and a private key. The private key is used to generate a digital signature, and the public key is used to verify the legitimacy of the signature.

[0082] Furthermore, in the process of obtaining financial data, there may be a situation where data cannot be successfully obtained. The reasons for the failure to obtain data include: first, invalid API key; the API interface call may fail because the provided API key is invalid or expired; second, invalid API request parameters; the provided API request parameters do not meet the requirements, such as incorrect parameter format, type or value range, which may also cause the interface call to fail; third, server failure or unavailability; API server failure, upgrade or unavailability may also cause the interface call to fail; fourth, interface restrictions or quota exceeded; the API interface may have access restrictions or quota restrictions, such as the number of calls per minute or hour, which will cause the interface call to fail when the limit is exceeded; fifth, network connection problems; network instability, delays or interruptions may also cause interface call failures; sixth, incorrect calling method; the API interface calling method is incorrect, such as using an unsupported HTTP method or an incorrect URL path, which leads to a call failure; seventh, security policy restrictions; the API interface may have a security policy, such as requiring an HTTPS encrypted connection or requiring specific request header information, and failure to meet the security policy requirements may cause the interface call to fail. The above reasons may lead to the failure of data acquisition. If there is a failure in actual application, the above reasons can be checked one by one.

[0083] See also Figure 4 Optionally, the step of generating a data form based on data display requirements includes:

[0084] Step S402: Select corresponding multiple fields according to data display requirements.

[0085] Step S404: Acquire the mapping relationship between the fields according to the asset structure information.

[0086] Step S406, designing a table structure according to the mapping relationship and constructing a data form.

[0087] Specifically, financial data is divided into different categories according to different display requirements. There are specific fields corresponding to data forms of different categories. In order to meet the display requirements of multiple users, the above fields are pre-configured for data forms of different categories in the embodiment of the present application. After receiving a user's display requirement, the corresponding field is searched according to the characteristics of the display requirement, and the corresponding data form is constructed according to the mapping relationship between the fields.

[0088] Furthermore, the mapping relationship represents the logical connection and correspondence between different fields. These relationships may be reflected in the correspondence between accounting subjects, such as the balance between asset subjects and liability subjects; the cross-checking relationship between financial statements, such as the data linkage between the balance sheet and the income statement; and the mapping relationship between financial data and business data. In addition, the fields of financial data are financial subjects. For example, when constructing an asset structure detail table, first obtain the associated fields such as bank deposits, cash, accounts receivable, etc. according to the asset structure information; when constructing an income statement, the relevant fields are operating income, operating costs, taxes and surcharges, etc. Then obtain the mapping relationship between the fields, for example, the existing assets are the sum of bank deposits, cash and accounts receivable; the profit is the operating income minus the operating costs, and then minus the taxes and surcharges.

[0089] The table structure is designed according to the mapping relationship, and the steps include determining the row and column layout, data labels, calculation formulas, etc. For example, in the income statement, operating income, operating costs, etc. can be used as row labels, and the data of each accounting period can be used as column labels.

[0090] In another feasible embodiment, multiple form templates may be pre-built according to display requirements, and the corresponding template may be called upon receiving specific display requirements.

[0091] The above-mentioned visual data processing method obtains the mapping relationship between each field in the financial data according to display needs, and then designs the table structure according to the mapping relationship, constructing the messy financial data into data forms of different themes, which is convenient for subsequent query and improves the query speed and efficiency.

[0092] See also Figure 5 Optionally, create an associated view based on the asset structure information between the financial data, and use the associated view to extract data from the data form, including:

[0093] Step S502: Write a query statement according to the mapping relationship and query logic.

[0094] Step S504: encapsulate the query statement into an associated view.

[0095] Step S506: In response to the query instruction, query the data form through the associated view.

[0096] Step S508: Use the query result as the data to be extracted.

[0097] Specifically, because the amount of financial data is large, the structure of the asset account table is complex, and there is a situation where one table corresponds to multiple tables, and the query time will be relatively long. Therefore, this application determines the selected fields in advance according to the mapping relationship, and then determines the query logic between the fields, and then encapsulates the written query statement into an associated view, so that the query result can be obtained by directly querying this associated view during the query. Exemplarily, for a user's personal information, his basic information is stored in a form, including gender, date of birth, ID number, address and telephone number, etc.; his education information is stored in another form, including academic qualifications and school, etc.; his work information is stored in another form, including company name, position, years of work, etc. By constructing an associated view, when the query instruction is to query the user's personal information, the data can be extracted from the above multiple forms and stored in the ODS layer of the data warehouse, providing a basis for subsequent data processing. Exemplarily, this application uses Kettle (Pentaho Data Integration) to perform data extraction operations.

[0098] The above visual data processing method, view query, improves query speed and efficiency. At the same time, users can only access the result set of the query they are allowed to query, which helps to limit access to specific rows and columns in the database. Different from directly managing permissions on the table, associated views can more finely control user access to data and increase data security.

[0099] Optionally, the extracted data is processed, including any one or more of data cleaning, data format conversion, data integration and standardization processing.

[0100] Specifically, there are many problems with the original data that need to be adjusted, such as data type mismatch, and the date field is in string format when it is passed in, which needs to be adjusted to date format. The obtained financial data has duplicates. If it is a complete duplicate row, it may be a problem with the data source, such as storing the same data in multiple systems at the same time, or an error when merging data. In this case, deduplication is required or there is a problem with the source table screening; if it is an incomplete duplicate row, there are conflicting data rows in different tables of the data warehouse, such as the same ID, but different contents of other fields. This may be due to different data sources, or changes in the same data source at different time points. In this case, it is necessary to check the data in the upstream system to confirm whether to merge. Furthermore, there may be problems with the quality of the obtained data not meeting the standards. This may be due to the need to set the length of the voucher summary field in order to adapt to data storage requirements, improve performance, save storage space, ensure data integrity, etc. In addition, when passing in data, sometimes the field length does not match, and the field length of the voucher summary field is too long when it is passed in, which will cause the obtained field data to be empty; when verifying the accuracy of the data, if empty data appears, consider appropriately lengthening the field length. For data performance issues, if the data loading process takes too long, you can add indexes or optimize the code to increase query speed and minimize subqueries. When the amount of data is too large, you can use the "WITH AS" statement, which helps improve query performance.

[0101] Exemplarily, the main steps of data cleaning include: removing duplicate data from the data to be displayed, and judging the field value by pre-setting judgment conditions, such as whether the field value is empty, whether the fixed field value length is correct, etc., such as the ID card number field, the fixed length is 18. In addition, if there are missing values ​​and incomplete data, the original file table of the ODS layer of the data warehouse is connected, and the matching and comparison are performed according to the database table name, and then filled and interpolated. The cleaned data is stored in the DWD layer of the data warehouse, providing a basis for subsequent data conversion and data integration.

[0102] Furthermore, since the data comes from various sources, the types of the data may be different and need to be converted into a unified format. For example, some amount data are integers and some are decimals, which need to be unified into decimals with two decimal places.

[0103] Optionally, sensitive data in financial data, such as name, password, phone number, address, bank card number, certificate number in personal information, business information credit code certificate, legal person, phone number in organizational information, and all relevant information of minors under the age of fourteen, need to be fully encrypted during the data conversion process. When using the encryption tool that comes with kettle, there will be problems such as encryption failure to decrypt and unsuccessful encryption parameter setting. For this, this application uses Java code to write an encryption program, pass in the plaintext data to be encrypted, convert the UTF-8 encoding to byte, and then use algorithms such as SM4, ECB or Padding to encrypt, return the ciphertext data in HEX format to implement encryption, compile the written code into a jar package, import the compiled jar package in the kettle program, and call the encryption method in the program to encrypt the data.

[0104] Furthermore, during the data conversion process, the application also lightly summarizes the data, merges multiple tables according to the fields associated with the tables, and merges or splits some fields. For example, the user's transaction data transfer and deposit are stored in the transfer table and deposit table respectively, and the two tables are associated according to the ID number and then merged. The data conversion and lightly summarized data are stored in the DWS layer of the data warehouse to provide a basis for subsequent data integration.

[0105] Furthermore, data integration is used to further aggregate the data conversion and lightly aggregated data into a summary table for display according to display requirements. It should be noted that each topic corresponds to a summary table.

[0106] According to the theme in the above step S204, each form is aggregated into a master table, which is stored in the ADS layer of the data warehouse to provide display data for the data screen.

[0107] The above-mentioned visual data processing method can make the displayed data more accurate, meet customer needs, and have good visual effects by cleaning, converting and integrating the data to be displayed. At the same time, the query scope of the data is narrowed by integrating and classifying, thereby improving the response efficiency.

[0108] Optionally, when using Kettle, data loss may occur when the amount of data is large, resulting in incomplete and inaccurate data. Therefore, this application uses the full load method for the first time, and the regular incremental update method is used later, that is, adding a time field when creating a data conversion. After the data loading and update is completed, the total amount of incremental data is queried by executing the SQL script to verify whether the data is lost. If lost, the database is rolled back and the data is reloaded and updated. Preventing data loss ensures the accuracy, real-time and consistency of the data.

[0109] Optionally, before storing the general table in the ADS layer of the data warehouse, the accuracy of the data is also confirmed. For example, given that the financial data are all accounts and amount values, the method is as follows: If there are sub-accounts under a specific account, compare the total amount of the sub-account with the total amount of the account. If they are consistent, the data is accurate. If there are no sub-accounts under a specific account, compare the values ​​of the same account between multiple tables. If they are consistent, the data is accurate. If a specific account in the entire table is unique, you can first use a standard financial calculation formula to verify the data. If the above verification is abnormal, call the FROM statement of the DWD detailed data layer to find the ODS original data layer data table corresponding to the account, prompt and report an error, and manually verify.

[0110] Optionally, there are multiple database forms in the target data warehouse, each of which is classified into multiple categories according to the subject, and the data to be displayed is loaded into the multiple database forms of the target data warehouse. Then the step of displaying the data to be displayed includes:

[0111] Setting key fields, where the key fields include one or more of table name, column name, creation time, and update time;

[0112] Query all database tables in the target data warehouse, extract key fields, and generate temporary tables based on the extracted key fields;

[0113] After receiving the query statement, the temporary table is queried, the corresponding database form is located according to the query result, and the data to be displayed in the database form is extracted for display.

[0114] Specifically, the key fields should be the iconic fields of the database form, that is, the temporary table stores the identification information of each database form. In this way, by querying the temporary table, the specific form or similar forms can be located according to the key fields therein, and then by filtering the subject, organization and date conditions, the specific data can be queried, thereby saving query time and improving query performance.

[0115] Optionally, the visual data processing method provided in the embodiment of the present application further includes:

[0116] The development trend of the data to be displayed after analysis and processing. If the development trend exceeds the preset threshold, a warning message will be displayed.

[0117] Specifically, this application uses Yager BI to create a data screen. Yager BI is a business intelligence (BI) product, a new generation of intelligent BI service platform tailored for cloud users with particularly large amounts of data. At the same time, it also supports offline users, that is, the data is not hosted on Alibaba Cloud, and it can be directly connected to the local Excel data source or database data source. Yager BI has powerful data processing, data analysis and visualization capabilities, and is an efficient analysis and presentation platform for big data.

[0118] Configure the data source in Yager BI, write SQL to create a data set for the data to be displayed, such as user regional distribution, extract only the address field in the user information; create a data screen, select relevant charts according to data characteristics, and configure the data in the data set; for example, use color maps with geographic attributes to present geographic locations and data, use month-on-month charts with percentages, and use bar charts for comparison data with years, etc. By selecting a suitable background image, reasonably matching and arranging charts, a large-screen display is formed.

[0119] Furthermore, in the process of displaying the above data, for charts that display data development trends, such as line charts, area charts, bar charts, step charts, slope charts, etc., if the data development trend exceeds a preset threshold, a warning message can also be displayed to alert users.

[0120] The above-mentioned visual data processing method loads the financial data obtained from multiple databases into a pre-generated data form, and uses the associated view to extract the required data from the data form, which improves the query speed and efficiency while also increasing the security of the data; finally, the extracted data is processed and loaded into the target data warehouse for users to view and display. This application can convert massive amounts of financial data into display data that meets the requirements according to user needs, providing users with display results with fast processing speed and high accuracy.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a visual data processing device for implementing the visual data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more visual data processing device embodiments provided below can refer to the limitations of the visual data processing method above, and will not be repeated here.

[0123] See also Figure 6In an exemplary embodiment, the present application provides a visual data processing device, including: an acquisition module, a data extraction module, a data processing module and a display module, wherein:

[0124] The acquisition module is used to acquire financial data and load it into a pre-generated data form, wherein the data form is generated based on data display requirements.

[0125] The data extraction module is used to create an associated view based on the asset structure information between financial data, and extract data from the data form using the associated view.

[0126] The data processing module is used to process the extracted data to obtain the data to be displayed.

[0127] The display module is used to load the data to be displayed into the target data warehouse for display.

[0128] Optionally, the steps before the acquisition module acquires the financial data include: acquiring document information of the data system to be accessed, performing interface encapsulation on the data system according to the document information; and acquiring the financial data from the data system after the interface encapsulation.

[0129] Optionally, the acquisition module is also provided with a timed task to acquire data from the data system at a fixed time every day, wherein the acquired financial data is signature-verified.

[0130] Optionally, the visual data processing device of the embodiment of the present application further includes: a table construction module. The step of generating a data form based on the data display requirements by the table construction module includes: selecting a plurality of corresponding fields according to the data display requirements, obtaining a mapping relationship between the fields according to the asset structure information, designing a table structure according to the mapping relationship, and constructing a data form.

[0131] Optionally, the data extraction module creates an associated view based on the asset structure information between the financial data, and uses the associated view to extract data from the data form, including: writing a query statement based on the mapping relationship and query logic; encapsulating the query statement into an associated view; querying the data form through the associated view in response to the query instruction; and using the query result as the extracted data.

[0132] Optionally, the data processing module processes the extracted data, including any one or more of data cleaning, data format conversion, data integration and standardization processing.

[0133] Optionally, the data processing module is further used to perform encryption operations on sensitive data after the data format is converted.

[0134] Optionally, the data processing module is also used to perform light aggregation on the data during the data conversion process.

[0135] Optionally, the data processing module is further used to analyze the development trend of the processed data to be displayed, and if the development trend exceeds a preset threshold, a warning message is displayed.

[0136] Optionally, the step of displaying the data to be displayed by the display module includes: setting key fields, wherein the key fields include one or more of table name, column name, creation time and update time; querying all database forms in the target data warehouse, extracting key fields, and generating a temporary table based on the extracted key fields; querying the temporary table after receiving the query statement, locating the corresponding database form according to the query result, and extracting the data to be displayed in the database form for display.

[0137] The above-mentioned visual data processing device loads the financial data obtained from multiple databases into a pre-generated data form, and uses the associated view to extract the required data from the data form, which improves the query speed and efficiency while also increasing the security of the data; finally, the extracted data is processed and loaded into the target data warehouse for users to view and display. This application can convert massive amounts of financial data into display data that meets the requirements according to user needs, providing users with display results with fast processing speed and high accuracy.

[0138] Each module in the above-mentioned visualization data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0139] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a visual data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0140] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method steps in the above embodiment when executing the computer program.

[0142] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method steps in the above embodiment are implemented.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0145] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A visual data processing method, characterized in that: The method comprises: Obtain financial data and load it into a pre-generated data form, wherein the data form is generated based on data display requirements; Creating an associated view based on the asset structure information between the financial data, and extracting data from the data form using the associated view; Process the extracted data to obtain the data to be displayed; The data to be displayed is loaded into the target data warehouse for display.

2. The method according to claim 1, characterized in that The steps before obtaining the financial data include: Acquire document information of a data system to be accessed, and perform interface encapsulation on the data system according to the document information; Financial data is obtained from the data system after interface encapsulation.

3. The method according to claim 1, characterized in that The steps to generate a data form based on data display requirements include: Select multiple corresponding fields according to data display requirements; Obtain the mapping relationship between various fields based on asset structure information; A table structure is designed according to the mapping relationship to construct the data form.

4. The method according to claim 1, characterized in that: The step of creating an associated view based on the asset structure information between the financial data and extracting data from the data form using the associated view includes: Acquire a mapping relationship between the fields according to the asset structure information between the financial data, and determine a query logic based on the mapping relationship; Write a query statement according to the mapping relationship and query logic; Encapsulating the query statement as an associated view; In response to a query instruction, query the data form through the associated view; The query result is used as the data to be extracted.

5. The method according to claim 1, characterized in that The processing of the extracted data includes: any one or more of data cleaning, data conversion and data integration.

6. The method according to claim 5, characterized in that The processing of the extracted data also includes: encrypting the sensitive data during the data conversion process.

7. The method according to any one of claims 1 to 6, characterized in that The data to be displayed is loaded into a plurality of database tables of the target data warehouse; Displaying the data to be displayed includes: Setting key fields, wherein the key fields include one or more of table name, column name, creation time and update time; Query all database tables in the target data warehouse, extract key fields, and generate temporary tables based on the extracted key fields; After receiving the query statement, the temporary table is queried, the corresponding database form is located according to the query result, and the data to be displayed in the database form is extracted for display.

8. A visual data processing device, characterized in that: The device comprises: An acquisition module, used to acquire financial data and load it into a pre-generated data form, wherein the data form is generated based on data display requirements; A data extraction module, used to create an associated view based on the asset structure information between the financial data, and extract data from the data form using the associated view; A data processing module is used to process the extracted data to obtain data to be displayed; The display module is used to load the data to be displayed into the target data warehouse for display.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images