Data visualization processing method and system, terminal and storage medium

Abstract

The invention discloses a data visualization processing method and system, a terminal and a storage medium. The method comprises the steps of collecting multi-source heterogeneous data of a target object based on a Python module; importing the multi-source heterogeneous data into a data warehouse, and performing data cleaning and conversion on the multi-source heterogeneous data according to a preset data cleaning rule and a conversion strategy to obtain target data; and on the basis of a DataBorad tool, presenting the target data to a user in a visual form through a built-in data visualization function of the DataBorad tool. According to the method, various data in the research and development process are systematically collected, sorted and deeply analyzed, and the data analysis result is visually displayed to the user, so that the readability and usability of the data are greatly improved, the user can quickly understand the research and development quality condition conveniently, accurate and powerful data support is provided for an enterprise decision-making layer, and the research and development efficiency is improved. And scientific and reasonable research and development decisions and quality optimization work can be made.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data visualization processing method, system, terminal and computer-readable storage medium. Background Art

[0002] In today's fiercely competitive market, product R&D quality has become a key factor in companies' success. This quality not only impacts a product's competitiveness in the market but also directly impacts a company's reputation and sustainable development.

[0003] The traditional R&D model has many shortcomings in quality control and risk response, and it is difficult to meet the growing high-quality R&D needs of enterprises. That is, the analysis quality of enterprise R&D data by existing technologies is not high, and the data analysis results obtained cannot provide effective R&D reference value for enterprises.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a data visualization processing method, system, terminal and computer-readable storage medium, aiming to solve the problem that the analysis quality and efficiency of enterprise R&D data in the existing technology are low, and the data analysis results obtained cannot provide effective R&D reference value for the enterprise.

[0006] To achieve the above object, the present invention provides a data visualization processing method, which includes the following steps:

[0007] Collect multi-source heterogeneous data of target objects based on Python modules;

[0008] Importing the multi-source heterogeneous data into a data warehouse, and performing data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data;

[0009] Based on the DataBorad tool, the target data is presented to the user in a visual form through the built-in data visualization function of the DataBorad tool.

[0010] Optionally, in the data visualization processing method, the multi-source heterogeneous data includes code submission volume, test data, and user feedback data.

[0011] Optionally, in the data visualization processing method, the code submission amount represents the number of lines of code modified by each developer;

[0012] The test data represents the number of bugs, test cases, and automated tests found by testers before release.

[0013] The user feedback data represents various issues reported by actual users after the product is officially released.

[0014] Optionally, in the data visualization processing method, the data cleaning includes noise data removal and erroneous data correction.

[0015] Optionally, the data visualization processing method, wherein the multi-source heterogeneous data is imported into a data warehouse, and the multi-source heterogeneous data is cleaned and transformed according to preset data cleaning rules and transformation strategies to obtain target data, specifically includes:

[0016] Importing the collected multi-source heterogeneous data into a data warehouse, and in the data warehouse, using a data cleaning tool to remove noise data and correct erroneous data according to preset data cleaning rules to obtain intermediate multi-source heterogeneous data;

[0017] According to the conversion standard table, the intermediate multi-source heterogeneous data is formatted using a data cleaning tool or a Python data processing module, and the intermediate multi-source heterogeneous data is converted into a standard data table to obtain the target data and store it.

[0018] Optionally, in the data visualization processing method, the visualization form includes: charts and interactive dashboards.

[0019] Optionally, in the data visualization processing method, the target data is used to reflect the R&D quality status of the target object.

[0020] In addition, to achieve the above-mentioned purpose, the present invention further provides a data visualization processing system, wherein the data visualization processing system includes:

[0021] Data acquisition module, used to collect multi-source heterogeneous data of target objects based on Python modules;

[0022] A data processing module is used to import the multi-source heterogeneous data into a data warehouse, and perform data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data;

[0023] The visualization display module is used to present the target data to the user in a visual form based on the DataBorad tool through the built-in data visualization function of the DataBorad tool.

[0024] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a data visualization processing program stored on the memory and runnable on the processor, and when the data visualization processing program is executed by the processor, the steps of the data visualization processing method described above are implemented.

[0025] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data visualization processing program, and when the data visualization processing program is executed by a processor, the steps of the data visualization processing method described above are implemented.

[0026] In the present invention, multi-source heterogeneous data of the target object is collected based on the Python module; the multi-source heterogeneous data is imported into the data warehouse, and according to the preset data cleaning rules and transformation strategies, the multi-source heterogeneous data is cleaned and transformed to obtain the target data; based on the DataBorad tool, the target data is presented to the user in a visual form through the built-in data visualization function of the DataBorad tool. The present invention greatly improves the readability and ease of use of the data by systematically collecting, organizing and deeply analyzing various types of data in the R&D process, and presenting the data analysis results to the user in a visual manner, making it convenient for users to quickly understand the R&D quality status, providing accurate and powerful data support for the decision-makers of the enterprise, and assisting them in making scientific and reasonable R&D decisions and quality optimization work. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a preferred embodiment of the data visualization processing method of the present invention;

[0028] Figure 2 Schematic diagram of the entire data processing process in a preferred embodiment of the data visualization processing method of the present invention;

[0029] Figure 3 It is a structural diagram of a preferred embodiment of the data visualization processing system of the present invention;

[0030] Figure 4 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] The data visualization processing method described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2As shown, the data visualization processing method includes the following steps:

[0033] Step S10: Collect multi-source heterogeneous data of the target object based on the Python module.

[0034] Specifically, Python is a high-level, interpreted, general-purpose programming language. It is known for its concise, easy-to-read syntax and powerful functions. It is widely used in web development, data analysis, artificial intelligence, scientific computing, automated scripting and other fields, especially data cleaning, visualization, and statistical analysis. It is characterized by high development efficiency, small amount of code, rich community resources, low learning cost and strong compatibility.

[0035] The target objects of the present invention mainly refer to enterprises. According to the R&D status of the enterprises, it is necessary to collect multi-source heterogeneous data of the target enterprises through the Python module, that is, to collect multi-source heterogeneous data such as code submission volume, test data, user feedback data, etc. generated in the R&D process through the Python module. The code submission volume represents the number of lines of code modified by each developer (the frequency of code changes by developers in the version control system (such as Git) is counted to reflect the team development activity), the test data represents the number of bugs, test cases, and automated tests found by testers before the release, and the user feedback data represents various problems reported by actual users after the product is officially released, including requirements, bugs, optimization, etc. Various modules are required for use in the Python language, such as database connection modules and git modules. The common services are encapsulated and can be called by direct reference, which is convenient, fast and highly reusable. Therefore, the present invention uses specific Python modules to achieve accurate capture and efficient collection of data.

[0036] Step S20: import the multi-source heterogeneous data into a data warehouse, and perform data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data.

[0037] Specifically, Data Warehouse (DW / DWH, i.e. Figure 2 A data warehouse is a subject-oriented, integrated, relatively stable data collection that reflects historical changes and is used to support enterprise decision-making analysis. Its core goal is to integrate data scattered across different systems, provide a unified analytical view, and assist business intelligence (BI) and data analysis.

[0038] The collected multi-source heterogeneous data is imported into a data warehouse. In the data warehouse, according to the preset data cleaning rules (such as missing value processing, outlier processing, duplicate data processing, format standardization, data consistency verification, text data cleaning), the multi-source heterogeneous data is cleaned using a data cleaning tool. The data cleaning includes noise data removal and error data correction to obtain intermediate multi-source heterogeneous data. For example, in conventional development, there are many invalid codes such as comments, deletions, compiled dis files, etc. in the data submitted by the developer, which are not valid codes. These are all processed by cleaning. According to the conversion standard table (i.e., conversion strategy), the intermediate multi-source heterogeneous data is converted into a standard data table (converted into a standard data table to provide standard data for subsequent analysis), and the target data is obtained and stored. For example, the intermediate multi-source heterogeneous data is converted into a two-dimensional data table with a standard format, such as a table that records the amount of code submitted by the personnel. The fields basically have information such as the submitter, submission time, submission project, total submitted code amount, and valid code amount.

[0039] Step S30: Based on the DataBorad tool, the target data is presented to the user in a visual form through the built-in data visualization function of the DataBorad tool.

[0040] Specifically, DataBoard generally refers to a category of data visualization and business intelligence (BI) tools used to connect to data sources, create interactive dashboards, and support data analysis. DataBoard uses built-in data visualization capabilities to present target data (analysis results) processed by the data warehouse (i.e., target data) to users in the form of intuitive charts and interactive dashboards. This greatly improves data readability and usability, allowing users to quickly understand R&D quality status and facilitate R&D decision-making and quality optimization.

[0041] This invention leverages advanced data analysis methods to deeply explore the value of R&D process data, striving to comprehensively improve product quality and effectively reduce R&D risks. By systematically collecting, organizing, and deeply analyzing various types of data during the R&D process, it provides accurate and powerful data support to enterprise decision-makers, assisting them in making scientific and reasonable R&D decisions. At the same time, it innovatively introduces cutting-edge technologies such as automated testing and intelligent analysis to fundamentally break through traditional R&D efficiency and quality bottlenecks, significantly improving R&D efficiency and optimizing product quality.

[0042] Further, if Figure 3As shown, based on the above data visualization processing method, the present invention also provides a data visualization processing system, wherein the data visualization processing system includes:

[0043] A data collection module 51 is used to collect multi-source heterogeneous data of a target object based on a Python module;

[0044] The data processing module 52 is used to import the multi-source heterogeneous data into the data warehouse, and perform data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data;

[0045] The visualization display module 53 is used to present the target data to the user in a visual form based on the DataBorad tool through the built-in data visualization function of the DataBorad tool.

[0046] Further, if Figure 4 As shown, based on the above-mentioned data visualization processing method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0047] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a data visualization processing program 40 is stored on the memory 20, and the data visualization processing program 40 can be executed by the processor 10, thereby realizing the data visualization processing method in the present application.

[0048] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes stored in the memory 20 or process data, such as executing the data visualization processing method.

[0049] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The processor 10, memory 20, and display 30 of the terminal communicate with each other via a system bus.

[0050] In one embodiment, when the processor 10 executes the data visualization processing program 40 in the memory 20, the following steps are implemented:

[0051] Collect multi-source heterogeneous data of target objects based on Python modules;

[0052] Importing the multi-source heterogeneous data into a data warehouse, and performing data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data;

[0053] Based on the DataBorad tool, the target data is presented to the user in a visual form through the built-in data visualization function of the DataBorad tool.

[0054] The multi-source heterogeneous data includes code submission volume, test data, and user feedback data.

[0055] The code submission amount represents the number of lines of code modified by each developer;

[0056] The test data represents the number of bugs, test cases, and automated tests found by testers before release.

[0057] The user feedback data represents various issues reported by actual users after the product is officially released.

[0058] The data cleaning includes noise data removal and error data correction.

[0059] The step of importing the multi-source heterogeneous data into a data warehouse and performing data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data specifically includes:

[0060] Importing the collected multi-source heterogeneous data into a data warehouse, and in the data warehouse, using a data cleaning tool to remove noise data and correct erroneous data according to preset data cleaning rules to obtain intermediate multi-source heterogeneous data;

[0061] According to the conversion standard table, the intermediate multi-source heterogeneous data is formatted using a data cleaning tool or a Python data processing module, and the intermediate multi-source heterogeneous data is converted into a standard data table to obtain the target data and store it.

[0062] The visualization forms include: charts and interactive dashboards.

[0063] The target data is used to reflect the R&D quality status of the target object.

[0064] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data visualization processing program, and when the data visualization processing program is executed by a processor, the steps of the data visualization processing method described above are implemented.

[0065] In summary, the present invention provides a data visualization processing method, system, terminal and computer-readable storage medium, the method comprising: collecting multi-source heterogeneous data of a target object based on a Python module; importing the multi-source heterogeneous data into a data warehouse, and performing data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data; based on the DataBorad tool, the target data is presented to the user in a visual form through the built-in data visualization function of the DataBorad tool. The present invention greatly improves the readability and ease of use of the data by systematically collecting, organizing and deeply analyzing various types of data in the R&D process, and presenting the data analysis results to the user in a visual manner, making it convenient for users to quickly understand the R&D quality status, and providing accurate and powerful data support for enterprise decision-makers, assisting them in making scientific and reasonable R&D decisions and quality optimization work.

[0066] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0067] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0068] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A data visualization processing method, characterized in that: The data visualization processing method includes: Collect multi-source heterogeneous data of target objects based on Python modules; Importing the multi-source heterogeneous data into a data warehouse, and performing data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data; Based on the DataBorad tool, the target data is presented to the user in a visual form through the built-in data visualization function of the DataBorad tool.

2. The data visualization processing method according to claim 1, characterized in that: The multi-source heterogeneous data includes code submission volume, test data, and user feedback data.

3. The data visualization processing method according to claim 2, characterized in that: The code commit amount represents the number of lines of code modified by each developer; The test data represents the number of bugs, test cases, and automated tests found by testers before release. The user feedback data represents various issues reported by actual users after the product is officially released.

4. The data visualization processing method according to claim 1, characterized in that: The data cleaning includes noise data removal and error data correction.

5. The data visualization processing method according to claim 4, characterized in that: The step of importing the multi-source heterogeneous data into a data warehouse and performing data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data specifically includes: Importing the collected multi-source heterogeneous data into a data warehouse, and in the data warehouse, using a data cleaning tool to remove noise data and correct erroneous data according to preset data cleaning rules to obtain intermediate multi-source heterogeneous data; According to the conversion standard table, the intermediate multi-source heterogeneous data is formatted using a data cleaning tool or a Python data processing module, and the intermediate multi-source heterogeneous data is converted into a standard data table to obtain the target data and store it.

6. The data visualization processing method according to claim 1, characterized in that: The visualization forms include: charts and interactive dashboards.

7. The data visualization processing method according to claim 1, characterized in that: The target data is used to reflect the R&D quality status of the target object.

8. A data visualization processing system, characterized in that: The data visualization processing system includes: Data acquisition module, used to collect multi-source heterogeneous data of target objects based on Python modules; A data processing module is used to import the multi-source heterogeneous data into a data warehouse, and perform data cleaning and transformation on the multi-source heterogeneous data according to preset data cleaning rules and transformation strategies to obtain target data; The visualization display module is used to present the target data to the user in a visual form based on the DataBorad tool through the built-in data visualization function of the DataBorad tool.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a data visualization processing program stored in the memory and executable on the processor. When the data visualization processing program is executed by the processor, the steps of the data visualization processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a data visualization processing program, and when the data visualization processing program is executed by a processor, the steps of the data visualization processing method according to any one of claims 1 to 7 are implemented.

Images