Data processing method and device based on SaaS platform, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and provides a data processing method and device based on an SaaS platform and a storage medium, and the method comprises the steps: collecting data from different data sources through a unified data collection interface provided by the SaaS platform; automatically identifying and converting data acquired from different data sources to obtain target format data; storing the target format data to a cloud database by adopting a distributed storage scheme; analyzing the target format data stored in the cloud database based on a configurable data processing pipeline provided by the SaaS platform; and displaying the analysis result of the target format data to a user in a visual mode. According to the technical scheme provided by the invention, data with relatively large data volume can be flexibly and efficiently processed at low cost.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment and computer storage medium based on a SaaS platform. Background Art

[0002] Software-as-a-Service (SaaS) is a model for delivering software over the internet. Vendors deploy application software on their own servers. Customers can order the required application software services from the vendors over the internet based on their specific needs, paying the vendors based on the amount and duration of the services ordered. In contrast to SaaS, many enterprises still rely on traditional data processing methods, such as deploying software locally for data processing. However, this approach has drawbacks such as high cost, limited flexibility, and poor scalability. Summary of the Invention

[0003] The present application provides a data processing method, apparatus, device and computer storage medium based on a SaaS platform, which can process large amounts of data flexibly and efficiently at low cost.

[0004] In one aspect, the present application provides a data processing method based on a SaaS platform, the method comprising:

[0005] Collect data from different data sources through the unified data collection interface provided by the SaaS platform;

[0006] Automatically identifying and converting the data collected from different data sources to obtain target format data;

[0007] Storing the target format data in a cloud database using a distributed storage solution;

[0008] Analyze the target format data stored in the cloud database based on the configurable data processing pipeline provided by the SaaS platform;

[0009] The analysis results of the target format data are displayed to the user in a visual manner.

[0010] On the other hand, the present application provides a data processing device based on a SaaS platform, the device comprising:

[0011] The collection module is used to collect data from different data sources through the unified data collection interface provided by the SaaS platform;

[0012] A conversion module, configured to automatically identify and convert the data collected from different data sources to obtain target format data;

[0013] A storage module, configured to store the target format data in a cloud database using a distributed storage solution;

[0014] An analysis module, configured to analyze the target format data stored in the cloud database based on a configurable data processing pipeline provided by the SaaS platform;

[0015] The display module is used to display the analysis results of the target format data to the user in a visual manner.

[0016] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the technical solution of the data processing method based on the SaaS platform are implemented.

[0017] In a fourth aspect, the present application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the technical solution of the data processing method based on the SaaS platform as described above.

[0018] From the technical solution provided by the present application, it can be seen that after collecting data from different data sources through the unified data collection interface provided by the SaaS platform, the data collected from different data sources are automatically identified and converted to obtain target format data; a distributed storage solution is used to store the target format data in the cloud database; based on the configurable data processing pipeline provided by the SaaS platform, the target format data stored in the cloud database is analyzed; and the analysis results of the target format data are displayed to the user in a visual manner. On the one hand, since the above technical solution is implemented based on the SaaS platform, it can reduce the purchase and maintenance costs of hardware and software through the sharing and on-demand use of cloud resources; on the other hand, the above technical solution is adjusted and configured according to different business needs, providing an extensible processing architecture that can adapt to various changes in business needs, and the use of distributed storage and cloud computing technology can effectively process large amounts of data and ensure the stability and efficiency of the system when the data increases. In summary, the technical solution of the present application can process large amounts of data flexibly and efficiently at low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a flow chart of a data processing method based on a SaaS platform provided in an embodiment of the present application;

[0021] Figure 2 This is a structural diagram of a data processing device based on a SaaS platform provided in an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of the structure of the device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0026] Software-as-a-Service (SaaS) is a model for delivering software over the internet. Vendors deploy application software on their own servers. Customers can order the required application software services from the vendors over the internet based on their specific needs, paying the vendors based on the amount and duration of the services ordered. In contrast to SaaS, many enterprises still rely on traditional data processing methods, such as deploying software locally for data processing. However, this approach has drawbacks such as high cost, limited flexibility, and poor scalability.

[0027] In response to the above problems, the embodiments of the present application provide a data processing method, apparatus, device and computer storage medium based on a SaaS platform, which can process large amounts of data flexibly and efficiently at low cost.

[0028] The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] In this specification, for the convenience of description, the sizes of various parts shown in the drawings are not drawn according to the actual proportions.

[0030] As attached Figure 1 As shown, this application proposes a flow chart of a data processing method based on a SaaS platform, which mainly includes steps S101 to S105, as detailed below:

[0031] Step S101: Collect data from different data sources through the unified data collection interface provided by the SaaS platform.

[0032] It is understood that the data sources of the embodiments of the present application are diverse and may include databases, file systems, and hardware such as sensors. In principle, any source that generates data can serve as a data source for the embodiments of the present application. The aforementioned databases, file systems, and hardware such as sensors are merely examples of data sources and are not intended to be limiting.

[0033] Step S102: automatically identifying and converting data collected from different data sources to obtain target format data.

[0034] Since the data collected from different data sources are extensive and messy, such as databases, files, APIs, sensors, etc., these data usually use different data formats (such as JSON, XML, CSV, SQL, etc.). If the data collected from different data sources are used directly, on the one hand, the inconsistency of these data increases the complexity of subsequent processing and analysis; on the other hand, the data collected from different data sources may be noisy, and different systems or applications may require data in a specific format (for example, a machine learning model may require numerical data, while a reporting system may require structured tabular data). Directly using data collected from different data sources increases the difficulty of analyzing data for these systems or applications. Based on the above facts, in an embodiment of the present application, the data collected from different data sources are automatically identified and converted to obtain target format data. As an embodiment of the present application, the data collected from different data sources are automatically identified and converted to obtain target format data. This can be achieved by steps S1021 to S1023, as detailed below:

[0035] Step S1021: extracting metadata of data collected from different data sources.

[0036] The so-called metadata is data that describes data, including information such as the structure, format, and type of the data. Therefore, the metadata of data collected from different data sources is data that describes the data collected from different data sources. For example, the metadata of a CSV file may include column names and data types (strings, integers, floating-point numbers, etc.). Metadata helps understand the structure and content of data collected from different data sources, such as table column names, data types, relationships, etc. During the data conversion process, metadata is used to automatically map source data and target data structures, reducing manual configuration. In addition, metadata provides data quality information to help identify data problems, such as missing values ​​or inconsistent data. In an embodiment of the present application, metadata of data collected from different data sources can be collected using a data directory service or metadata management tool. Different methods can be used to extract metadata based on the source or type of data. Specifically, for relational databases, SQL queries such as `INFORMATION_SCHEMA` can be used to extract metadata such as table structure and column information; for files such as CSV, metadata of these CVS files can be extracted by extracting information such as field names and data types through file header information; for API data, its data structure information can be obtained through API documentation or by calling API interfaces to extract metadata, and so on.

[0037] Step S1022: converting the data collected from different data sources into standardized data according to the metadata of the data collected from different data sources.

[0038] As mentioned above, data is collected from different data sources through the unified data collection interface provided by the SaaS platform, that is, the data may come from multiple different sources, such as databases, files, APIs, sensors, etc. These sources usually use different data formats (such as JSON, XML, CSV, SQL, etc.). Since standardized formats help ensure the consistency of data, subsequent processing and analysis are more reliable. For example, the date format is unified into YYYY-MM-DD to avoid parsing errors caused by different formats; using standardized data formats can simplify the complexity in the data processing process and reduce redundant work in the processing steps; further, when data from multiple data sources need to be integrated together, the standardized format can greatly simplify the data integration work and ensure that the data can be seamlessly combined. Therefore, in an embodiment of the present application, the data collected from different data sources can be converted into standardized data based on the metadata of the data collected from different data sources. Specifically, it can be: analyzing the data formats of different data sources (such as CSV, JSON, XML, SQL, etc.) and determining the standardized target format, and then defining data conversion rules through configuration files or visualization tools, including data type mapping, field conversion, data cleaning, etc.; finally, according to the defined data conversion rules, the data collected from different data sources are converted into standardized data.

[0039] Step S1023: Based on business requirements, the standardized data is further converted into target format data.

[0040] Simply converting data collected from different data sources into standardized data is not enough to facilitate subsequent data processing. This is because, on the one hand, different systems or applications may require data in specific formats. For example, machine learning models may require numeric data, while reporting systems may require structured tabular data. On the other hand, ensuring data interoperability and compatibility is crucial for distributed systems or cross-organizational data sharing. For example, for e-commerce platforms, converting the age field in user data from text (such as "25 years") to numeric (such as 25) based on business needs facilitates subsequent data processing. Similarly, in the data warehouse field, aggregating sales data by quarter based on analytical needs also facilitates subsequent data processing. And so on. Specifically, according to business needs, further converting standardized data into target format data can be: according to business needs, aggregating or grouping standardized data, and then converting the standardized data into the format required by the target system or application, wherein, according to business needs, aggregating or grouping standardized data includes time dimension aggregation, that is, aggregating standardized data by day, week, month, quarter or year, and classification dimension aggregation, that is, aggregating standardized data by product category, region and other dimensions, etc.; and converting standardized data into the format required by the target system or application can specifically be structured data conversion, that is, converting data in JSON or XML format into a table format in a relational database, classification variable conversion, that is, converting string classification variables into numerical encoding and numerical calculation, that is, performing necessary numerical calculations or conversions, etc.

[0041] As can be seen from the above examples, standardizing and converting the data format collected from data sources is a key step in ensuring data quality and consistency. This helps address the diversity of data sources, simplifies data processing, meets the needs of target systems or applications, improves the efficiency of data processing and analysis, and ensures data interoperability across different systems and applications. These processes can better support data-driven decision-making and business processes.

[0042] Step S103: Using a distributed storage solution to store the target format data in a cloud database.

[0043] Compared to traditional storage technologies, distributed storage solutions offer significant advantages in scalability, high availability, load balancing, and data security. Therefore, after automatically identifying and converting data collected from different data sources to obtain target format data, a distributed storage solution can be used to store the target format data in a cloud database. Specifically, as one embodiment of the present application, using a distributed storage solution to store target format data in a cloud database can be achieved through steps S1031 to S1033, as detailed below:

[0044] Step S1031: determining a data sharding strategy for sharding the target format data according to the characteristics of the target format data.

[0045] In an embodiment of the present application, in order to cope with the expansion of data, such as similar table partitioning operations in a relational database, data sharding is performed, that is, global data is divided into independent and orthogonal data subsets according to certain rules, so that these data subsets can be distributed to different nodes for storage, which can improve data availability, query efficiency, achieve load balancing, increase the locality of transaction processing, etc. Data sharding is essentially a way to optimize the data storage structure, so that data can be managed and accessed more efficiently at the storage level. After data sharding, each fragment can be stored in more than one place (server), and different shards are stored in different storage locations, thereby changing the storage layout of the data.

[0046] The data sharding strategy depends on characteristics such as the size of the data to be sharded, the access frequency (which data is frequently accessed and which data is rarely accessed), and the correlation (which data is frequently accessed together). Therefore, the data sharding strategy for sharding the target format data can be determined based on the characteristics of the target format data. Specifically, if the target format data is high-frequency query data or high-write load data, a data sharding strategy based on row sharding is determined for the target format data of high-frequency query data or high-write load data; if the target format data is low-frequency query data or low-write load data, a data sharding strategy based on column sharding is determined for the target format data of low-frequency query data or low-write load data; if the target format data is strongly associated data, a data sharding strategy based on mixed row and column sharding is determined for the target format data of strongly associated data. In short, for target format data with large data volume and frequent queries, a data sharding strategy based on row sharding can be adopted; for target format data with complex table structure and too many columns, a data sharding strategy based on column sharding can be adopted; and for target format data with complex and high concurrency, a data sharding strategy based on mixed row and column sharding can be adopted.

[0047] Step S1032: fragment the target format data according to the data fragmentation strategy.

[0048] Taking the case where the target format data belongs to a relational database as an example, if it is decided to adopt a data sharding strategy based on row sharding, then sharding the target format data according to the data sharding strategy can be specifically as follows: selecting a sharding key, that is, determining a column as the sharding key, such as user ID, order ID, etc., and then creating a partitioned table, that is, using SQL commands to create a horizontally partitioned table, thereby sharding the target format data. Taking the case where the target format data belongs to a distributed database as an example, if it is decided to adopt a data sharding strategy based on column sharding, then sharding the target format data according to the data sharding strategy can be specifically as follows: analyzing column usage, that is, by analyzing query patterns, finding out which columns are frequently queried together and which columns can be stored separately (for example, in a user table, basic user information and detailed user behavior information can be separated), and then splitting the table structure, that is, splitting the original table by column, thereby sharding the target format data. If you decide to adopt a data sharding strategy based on mixed row and column sharding, then sharding the target format data according to the data sharding strategy can be specifically as follows: split the table into different sub-tables based on column usage; after sharding based on columns, perform row-based sharding on each sub-table; configure sharding rules in the SQL partition table or distributed database to ensure that the sharding strategies of different sub-tables are independent, thereby sharding the target format data.

[0049] Step S1033: Based on data reliability requirements and / or access load, copy the sharded data and store the data copies in the cloud database of the storage node.

[0050] Data replication involves creating copies of data on different nodes, allowing the data in a database to be backed up to servers or sites connected to the internet, wide area network, or local area network. While data replication increases data redundancy in distributed storage solutions, it also prevents distributed databases from crashing due to a single node failure, improving data availability. For example, in a distributed database system, data may be replicated across multiple servers for storage. These replicas provide access to data when needed, and even if a storage node fails, other replicas can still ensure data availability and integrity. The embodiment of the present application can copy the sharded data and store the data copy in the cloud database of the storage node according to the data reliability requirements and / or access load. Specifically, if the target format data is high reliability data and high read load data, the master node processes the write operation of the copy process and the slave node processes the read operation of the copy process, copies the sharded data and stores the data copy in the cloud database of the storage node, and takes over the write operation of the master node when the master computing node fails; if the target format data is high access load data, the master computing node and the slave computing node evenly copy the sharded high access load data and store the data copy in the cloud database of the storage node. In the above embodiment, the so-called high reliability data refers to these data that need to be available at any time and can be quickly restored even in the event of a node failure, such as data in fields with high data security requirements such as finance and medical care. The high read load in the high read load data refers to the need to support a large number of concurrent read requests, such as product queries on e-commerce websites. In fact, for high-reliability data and high-read load data, the master node handles the write operations of the replication process and the slave node handles the read operations of the replication process. In one case, the master computing node is responsible for processing all write operations, and the slave computing node copies the data from the master computing node and shares the read requests. In another case, after the data is copied to multiple computing nodes, the master computing node handles the write operations and the slave computing node handles the read requests. When the master computing node fails, the slave computing node automatically elects a new master computing node. Since the data is copied to multiple nodes, the advantage of this situation is that even if some computing nodes fail, the data is still available. In the above embodiment, for the cloud database solution in which the master computing node and the slave computing node evenly copy the sharded high-access load data and store the data copies in the storage node, since all nodes can process read and write operations, all nodes evenly share the read and write requests. Although there is no single point of failure, there may be data conflicts and consistency issues. The solution is to synchronize data between nodes through the Gossip protocol and use timestamps or version numbers to resolve data conflicts and ensure data consistency.

[0051] It should be noted that in the above embodiment, the master computing node is the node responsible for processing all write operations and managing data replication, sharding and other operations. It is usually the main source of data. When there is a data update, the master computing node is updated first, and then the update is propagated to the slave computing node; the slave computing node is the node used to copy the data of the master computing node, mainly used for read operations to reduce the load on the master computing node. The data update is the responsibility of the master computing node and then synchronized to the slave computing node; and the storage node is any node responsible for storing data. A master compute node can be a special type of storage node that not only stores data but also manages data writes and other management tasks. For example, in the aforementioned embodiment where "the master node handles write operations during the replication process and the slave node handles read operations during the replication process, replicating the sharded data and storing the data copies in the storage node's cloud database. In the event of a master compute node failure, the slave node takes over the master node's write operations," the master compute node is not only a storage node but also responsible for data consistency and replication management. A slave compute node is also a type of storage node, primarily responsible for read operations and providing data redundancy. The slave compute node receives and stores data copies from the master node to balance load and improve system fault tolerance. In the embodiment where "the master and slave compute nodes evenly replicate sharded, high-access-load data and store data copies in the storage node's cloud database," all compute nodes are typically storage nodes, with each compute node equally responsible for storing and processing data. In short, "master compute node" and "slave compute node" are role definitions within a specific replication architecture, while "storage node" is a broader concept encompassing any node that stores data. In different replication strategies, both the master and slave compute nodes can be considered storage nodes, but they have different responsibilities and functions.

[0052] In an embodiment of the above method, the target format data is also backed up and disaster-recovered when a distributed storage solution is used to store the target format data in a cloud database. Specifically, the method may include: determining a data backup solution based on the generation period or change of the target format data; selecting a backup time window, and backing up the target format data in the time window according to the data backup solution to obtain backup data; storing the backup data in multiple locations or synchronizing the backup data to a remote site using a real-time data replication solution.

[0053] Step S104: Based on the configurable data processing pipeline provided by the SaaS platform, the target format data stored in the cloud database is analyzed.

[0054] In an embodiment of the present application, the configurable data processing pipeline provided by the SaaS platform refers to a flexible system architecture that allows users to customize and adjust the various steps and algorithms of data processing as needed. In this way, it can support users to select and combine different processing steps according to specific business needs to achieve personalized data processing processes. The configurable data processing pipeline provided by the SaaS platform decomposes the data processing process into multiple modules, each of which is responsible for a specific processing task, such as data acquisition, data cleaning, data conversion, data analysis, data storage, etc. Each module should have a standardized interface for easy combination, and at the same time define standardized interfaces between modules (such as API or data format) to ensure that the modules can interoperate. Furthermore, the configurable data processing pipeline also provides configuration management tools, including a graphical interface (i.e., providing a graphical configuration management tool so that users can visually configure the data processing process), a command line interface (i.e., providing a command line interface that allows advanced users or automated systems to configure and manage data processing pipelines through scripts), etc. As for analyzing the target format data stored in the cloud database, it can be a variety of analysis methods such as time series analysis and machine learning algorithms for different data types.

[0055] Step S105: presenting the analysis results of the target format data to the user in a visual manner.

[0056] To facilitate users' understanding of the analysis results of the target format data, the analysis results of the target format data are presented to users in a visual manner, including dynamic interactive dashboards, multidimensional data displays, and text visualization. Among them, dynamic interactive dashboards use front-end visualization libraries such as D3.js and Plotly to provide real-time updates and interactive functions, allowing users to click, zoom, and hover to view detailed information; multidimensional data display refers to the display of high-dimensional data and multivariate relationships through parallel coordinate graphs (ParallelCoordinates) and scatter matrix graphs (ScatterMatrix); and text visualization is aimed at text data, mainly displaying Word Cloud, topic modeling, and sentiment analysis results, combining color and size to highlight key topics and emotional tendencies in the text.

[0057] From the above attached Figure 1From the example of the data processing method based on the SaaS platform, it can be seen that after collecting data from different data sources through the unified data collection interface provided by the SaaS platform, the data collected from different data sources are automatically identified and converted to obtain target format data; a distributed storage solution is used to store the target format data in the cloud database; based on the configurable data processing pipeline provided by the SaaS platform, the target format data stored in the cloud database is analyzed; and the analysis results of the target format data are displayed to the user in a visual manner. On the one hand, since the above technical solution is implemented based on the SaaS platform, it can reduce the purchase and maintenance costs of hardware and software through the sharing and on-demand use of cloud resources; on the other hand, the above technical solution is adjusted and configured according to different business needs, providing an extensible processing architecture that can adapt to various changes in business needs, and the use of distributed storage and cloud computing technology can effectively process large amounts of data and ensure the stability and efficiency of the system when the data increases. In summary, the technical solution of the present application can process large amounts of data flexibly and efficiently at low cost.

[0058] Please see the attached Figure 2 , is a data processing device based on a SaaS platform provided in an embodiment of the present application. The device may include a collection module 201, a conversion module 202, a storage module 203, an analysis module 204, and a presentation module 205, as detailed below:

[0059] The collection module 201 is used to collect data from different data sources through the unified data collection interface provided by the SaaS platform;

[0060] The conversion module 202 is used to automatically identify and convert data collected from different data sources to obtain target format data;

[0061] Storage module 203, used to store the target format data in the cloud database using a distributed storage solution;

[0062] An analysis module 204 is configured to analyze the target format data stored in the cloud database based on a configurable data processing pipeline provided by the SaaS platform;

[0063] The display module 205 is used to display the analysis results of the target format data to the user in a visual manner.

[0064] From the above attached Figure 2It can be seen from the example of a data processing device based on a SaaS platform that after collecting data from different data sources through the unified data collection interface provided by the SaaS platform, the data collected from different data sources are automatically identified and converted to obtain target format data; a distributed storage solution is used to store the target format data in a cloud database; based on the configurable data processing pipeline provided by the SaaS platform, the target format data stored in the cloud database is analyzed; and the analysis results of the target format data are displayed to the user in a visual manner. On the one hand, since the above technical solution is implemented based on the SaaS platform, it can reduce the purchase and maintenance costs of hardware and software through the sharing and on-demand use of cloud resources; on the other hand, the above technical solution is adjusted and configured according to different business needs, providing an extensible processing architecture that can adapt to various changes in business needs, and the use of distributed storage and cloud computing technology can effectively process large amounts of data and ensure the stability and efficiency of the system when the data increases. In summary, the technical solution of the present application can process large amounts of data flexibly and efficiently at low cost.

[0065] Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and capable of running on the processor 30, such as a program of a data processing method based on a SaaS platform. When the processor 30 executes the computer program 32, the steps in the embodiment of the data processing method based on the SaaS platform are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of the modules / units in the above-mentioned device embodiments are realized, for example Figure 2 The functions of the collection module 201, the conversion module 202, the storage module 203, the analysis module 204 and the presentation module 205 are shown.

[0066] Exemplarily, the computer program 32 of the data processing method based on the SaaS platform mainly includes: collecting data from different data sources through the unified data collection interface provided by the SaaS platform; automatically identifying and converting the data collected from different data sources to obtain target format data; using a distributed storage solution to store the target format data in a cloud database; based on the configurable data processing pipeline provided by the SaaS platform, analyzing the target format data stored in the cloud database; and presenting the analysis results of the target format data to the user in a visual manner. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31 and executed by the processor 30 to complete the present application. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3. For example, the computer program 32 can be divided into the functions of an acquisition module 201, a conversion module 202, a storage module 203, an analysis module 204 and a display module 205 (modules in the virtual device). The specific functions of each module are as follows: the acquisition module 201 is used to collect data from different data sources through the unified data acquisition interface provided by the SaaS platform; the conversion module 202 is used to automatically identify and convert data collected from different data sources to obtain target format data; the storage module 203 is used to store the target format data in the cloud database using a distributed storage solution; the analysis module 204 is used to analyze the target format data stored in the cloud database based on the configurable data processing pipeline provided by the SaaS platform; the display module 205 is used to display the analysis results of the target format data to the user in a visual manner.

[0067] The electronic device 3 may include but is not limited to a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of electronic device 3 and does not constitute a limitation of electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, a computing device may also include input and output devices, network access devices, buses, etc.

[0068] The processor 30 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0069] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard drive or memory of the electronic device 3. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 3. Furthermore, the memory 31 can include both an internal storage unit of the electronic device 3 and an external storage device. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or is about to be output.

[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0071] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0072] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0073] In the embodiments provided in this application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0074] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0075] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0076] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program of the data processing method based on the SaaS platform can be stored in a storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments, that is, collecting data from different data sources through the unified data acquisition interface provided by the SaaS platform; automatically identifying and converting the data collected from different data sources to obtain target format data; using a distributed storage solution to store the target format data in a cloud database; based on the configurable data processing pipeline provided by the SaaS platform, analyzing the target format data stored in the cloud database; and displaying the analysis results of the target format data to the user in a visual manner. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form. Storage media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunications signals, and software distribution media. It should be noted that the content of storage media may be appropriately increased or decreased based on the requirements of legislation and patent practice within a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, storage media do not include electric carrier signals and telecommunications signals.

[0077] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application. The specific implementation methods described above further explain the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present invention.

Claims

1. A data processing method based on a SaaS platform, characterized in that: The method comprises: Collect data from different data sources through the unified data collection interface provided by the SaaS platform; Automatically identifying and converting the data collected from different data sources to obtain target format data; Using a distributed storage solution to store the target format data in a cloud database; Analyze the target format data stored in the cloud database based on the configurable data processing pipeline provided by the SaaS platform; The analysis results of the target format data are displayed to the user in a visual manner.

2. The data processing method based on the SaaS platform as claimed in claim 1, characterized in that: The data collected from different data sources are automatically identified and converted to obtain target format data, including: Extracting metadata of the data collected from different data sources; According to the metadata, converting the data collected from different data sources into standardized data; According to business requirements, the standardized data is further converted into target format data.

3. The data processing method based on the SaaS platform as claimed in claim 1, characterized in that: The adopting of a distributed storage solution to store the target format data in a cloud database includes: Determining a data sharding strategy for sharding the target format data according to the characteristics of the target format data; Slice the target format data according to the data slicing strategy; According to data reliability requirements and / or access load, the sharded data is replicated and the data copies are stored in a cloud database of a storage node.

4. The data processing method based on the SaaS platform as claimed in claim 3, characterized in that: The step of determining a data sharding strategy for sharding the target format data according to the characteristics of the target format data includes: If the target format data is high-frequency query data or high-write load data, determine to adopt a data sharding strategy based on row sharding for the target format data of the high-frequency query data or high-write load data; If the target format data is low-frequency query data or low-write load data, determine to use a data sharding strategy based on column sharding for the target format data of the low-frequency query data or low-write load data; If the target format data is strongly associated data, it is determined that a data sharding strategy based on row-column mixing is adopted for sharding the target format data of the strongly associated data.

5. The data processing method based on the SaaS platform as claimed in claim 3, characterized in that: The method of replicating the sharded data and storing the data replicas in a cloud database of a storage node according to data reliability requirements and / or access loads includes: If the target format data is high reliability data and high read load data, the master node processes the write operation of the replication process and the slave node processes the read operation of the replication process, replicates the sharded data and stores the data copy in the cloud database of the storage node, and when the master computing node fails, the slave node takes over the write operation of the master node; If the target format data is high access load data, the master computing node and the slave computing node evenly replicate the segmented high access load data and store the data copies in the cloud database of the storage node.

6. The data processing method based on the SaaS platform according to any one of claims 1 to 5, characterized in that: The method further comprises: When the target format data is stored in a cloud database using a distributed storage solution, the target format data is backed up and disaster-tolerant.

7. The data processing method based on the SaaS platform as claimed in claim 6, characterized in that: When the target format data is stored in the cloud database using a distributed storage solution, backing up and disaster recovery of the target format data includes: Determine a data backup plan according to the generation period or change of the target format data; Selecting a backup time window, backing up the target format data in the time window according to the data backup scheme, and obtaining backup data; Store the backup data in multiple locations or synchronize the backup data to a remote site using a real-time data replication solution.

8. A data processing device based on a SaaS platform, characterized in that: The device comprises: The collection module is used to collect data from different data sources through the unified data collection interface provided by the SaaS platform; A conversion module, used for automatically identifying and converting the data collected from different data sources to obtain target format data; A storage module, used to store the target format data in a cloud database using a distributed storage solution; An analysis module, used for analyzing the target format data stored in the cloud database based on a configurable data processing pipeline provided by the SaaS platform; The display module is used to display the analysis results of the target format data to the user in a visual manner.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images