System and method for cloud-based smart analysis and visual generation

Abstract

The embodiment of the invention relates to a system and method for cloud-based intelligent analysis and visualization generation. A data processing and analysis system optimizes resources for data storage and refresh events. A zoner module for a data analysis system may receive a first customer criterion and a first customer data set including tabular data and calculate a score for generating a zoning policy. The selected partition policy may be implemented to produce aggregated data that may be stored in the smart data mart. The partitions can then be accessed by a data visualization platform to be intelligent so as to make an intelligent, dynamic response to the user's data analysis and visualization generation requests. By providing synchronous partitioning and intelligent refreshing of data (especially big data), the data can be moved from the back end to the front end, the number of user clicks is minimum, and performance delay is minimum.

Description

Technical Field

[0001] The present disclosure relates generally to analysis of complex data in the cloud, and more particularly to efficient and intelligent partitioning of large amounts of data and generation of real-time visual insights. Background Art

[0002] Data analytics is a growing field that derives insights from raw information sources. Data analytics techniques can vary, but they are typically used to collect, process, and analyze data for human consumption and decision making. Data analytics can be used to reveal trends and indicators that are lost or "invisible" due to large amounts of unanalyzed data. However, using this information in this way can optimize processes and improve the overall efficiency of systems that would not otherwise be available.

[0003] Traditionally, many visual dashboards have focused primarily on providing high-level summaries such as segment overviews, cohort analysis, and other similar metrics. To gain a competitive advantage, stakeholders (including line managers) seek tools that reveal deeper analytical insights by drilling down into lower-level data in reports. However, in the context of ‘big data’ (with trillions of table rows and massive storage), visual report generation becomes lagging, especially when responding to front-end refresh requests. This creates unnecessarily high-latency dashboards and hinders access to important insights.

[0004] There is a need in the art for a system and method that addresses the above-mentioned shortcomings. Summary of the invention

[0005] A data processing and analysis system that optimizes resources for data storage and refresh events is disclosed. The proposed system and method describe a dynamic and automated process for generating partitioning strategies for managing and maintaining implementations of big data in the cloud. The system and method solve the above problems by automatically scoring and identifying the optimal partition size that best suits the user needs of a given project. In one example, data from multiple sources can be integrated in a cloud-agnostic and data format-agnostic process, allowing the system to create new metrics and establish a collection of relationships that support the construction of a tabular semantic data model that powers visualization tools. In some embodiments, the proposed data analysis framework employs a highly optimized and automated partition loading technique that involves segmenting data into smaller units. In addition, custom data refresh techniques can selectively refresh only necessary partitions, and coupled with intelligent data mart ("mart") capabilities, the required computation time is greatly reduced as redundancy is eliminated.

[0006] In one aspect, the present disclosure provides a computer-implemented method for processing and partitioning data for a cloud computing-based infrastructure. The method includes a first step of receiving a first customer criterion and a first customer data set including tabular data at a partitioner module of a data analysis system, and a second step of calculating a first size score of the tabular data at the partitioner module, the first size score identifying a first data increment that is most likely to meet the first customer criterion. In addition, a third step includes generating a first partitioning strategy based on the first size score at the partitioner module, and a fourth step includes implementing the first partitioning strategy at the data analysis system to generate a set of aggregated data in the tabular data, the sets including the first partition. The fifth step of the method includes storing the set data as part of an intelligent data mart in the data analysis system. The sixth step includes receiving a first user query via a first user interface associated with a data visualization platform, and the seventh step includes determining that the first user query is related to the first partition. In addition, the method includes an eighth step of retrieving the first partition from the intelligent data mart via the data visualization platform, and a ninth step of generating a first visualization based on the first partition for the data visualization platform and via a generation artificial intelligence (AI) module.

[0007] In another aspect, the present disclosure provides a non-transitory computer-readable medium storing software, the software including instructions executable by one or more computers, which, when executed, cause the one or more computers to process and partition data for a cloud computing-based infrastructure by performing the following: (1) receiving a first customer criterion and a first customer data set including tabular data at a partitioner module of a data analysis system; (2) calculating a first size score for the tabular data at the partitioner module, the first size score identifying a first data increment that is most likely to meet the first customer criterion; (3) generating a first partitioning strategy at the partitioner module based on the first size score; (4) implementing the first partitioning strategy at the data analysis system to generate a set of aggregated data in the tabular data, the set including a first partition; (5) storing the set of aggregated data as part of an intelligent data mart in the data analysis system; (6) receiving a first user query via a first user interface associated with a data visualization platform; (7) determining that the first user query is related to the first partition; (8) retrieving the first partition from the intelligent data mart via the data visualization platform; and (9) generating a first visualization based on the first partition for the data visualization platform and via a generative artificial intelligence (AI) module.

[0008] In yet another aspect, the present disclosure provides a system for generating an application architecture for a cloud computing-based infrastructure, the system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable when executed by the one or more computers to cause the one or more computers to: (1) receive a first customer criterion and a first customer data set comprising tabular data at a partitioner module of a data analysis system; (2) calculate a first size score for the tabular data at the partitioner module, the first size score identifying a first data increment that is most likely to meet the first customer criterion; (3) generate a first partitioning strategy at the partitioner module based on the first size score; (4) implement the first partitioning strategy at the data analysis system to generate a set of aggregated data in the tabular data, the set comprising a first partition; (5) store the set of aggregated data in the data analysis system as part of an intelligent data mart; (6) receive a first user query via a first user interface associated with a data visualization platform; (7) determine that the first user query is related to the first partition; (8) retrieve the first partition from the intelligent data mart via the data visualization platform; and (9) generate a first visualization based on the first partition for the data visualization platform and via a generative artificial intelligence (AI) module.

[0009] Other systems, methods, features and advantages of the present disclosure will be apparent to those skilled in the art after studying the following drawings and detailed description. All such additional systems, methods, features and advantages are intended to be included in this specification and the present disclosure, to be within the scope of the present disclosure, and to be protected by the appended claims.

[0010] Although various embodiments have been described, the description is intended to be illustrative rather than restrictive, and it will be clear to those of ordinary skill in the art that more embodiments and implementations are possible within the scope of the embodiments. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are possible. Unless otherwise specified, any feature or element of any embodiment may be used in combination with or in place of any other feature or element in any other embodiment.

[0011] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The disclosed embodiments, features, and elements may also be combined with any conventional features or elements to form a unique invention as defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other inventions to form another different invention as defined by the claims. Therefore, it will be understood that any feature shown and / or discussed in the present disclosure may be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except according to the attached claims and their equivalents. In addition, various modifications and changes may be made within the scope of the attached claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention may be better understood with reference to the following drawings and descriptions. The components in the drawings are not necessarily drawn to scale, but emphasis is placed on illustrating the principles of the present invention. In addition, in the drawings, the same reference numerals represent corresponding parts in different views.

[0013] Figure 1 is an overview of an embodiment of an intelligent data partitioning and analysis process;

[0014] Figure 2 is a high-level view of a data analysis system according to one embodiment;

[0015] Figure 3 is a schematic diagram depicting features of an intelligent data mart of a data analysis system according to an embodiment;

[0016] Figure 4 is a schematic diagram depicting a scoring process of a partitioner module of a data analysis system according to an embodiment;

[0017] Figure 5 Some examples of partitioning strategy generation that may be produced by a partitioner module according to an embodiment are presented;

[0018] Figure 6 is an example of a synchronous partition operation according to one embodiment;

[0019] Figure 7 is a schematic flow chart depicting a method of generating visualizations for a user dashboard according to one embodiment;

[0020] Figure 8 A diagram depicting an example environment and components that can be used to implement the systems and / or methods described herein; and

[0021] Fig. 9 is a flow chart presenting a method of processing and partitioning data according to one embodiment. DETAILED DESCRIPTION

[0022] Visual dashboards are an integral part of data analysis, especially when dealing with large amounts of data. Generally speaking, dashboards present information in the form of high-level summaries of smaller data that are continuously aggregated, providing a quick overview of key insights. However, for a dashboard to be most effective, it must take into account both the larger 'big picture' of data and the nuanced insights that can only be detected at a granular level. Because the design and production of conventional visual dashboards rely on aggregated data, detailed analysis and exploration of granular models based on aggregated data remain challenging. For example, conventional front-end dashboards do not allow users to drill down into the data and can only generate superficial analytical insights. In addition, such dashboards are unable to ingest the large-scale data that is becoming increasingly common. There are still frequent lags as traditional visualization and reporting tools try to process big data in response to front-end refresh operations. In addition, sensitive customer data such as personally identifiable information (PII) or protected health information (PHI) cannot be obtained without risk of exposure because the data must be entered into the reporting tool through import mode or direct query mode.

[0023] Despite this context, nearly every level of business decision is increasingly expected to be powered by data and, in some cases, artificial intelligence (AI)-driven insights. There is a pressing need for a no-code approach to data analysis and visualization dashboard generation that provides data security and enables users to perform efficient analysis without relying on coding expertise, thereby unlocking valuable data-driven insights that would otherwise be lost in massive data stores.

[0024] The proposed system and method are directed to the automation and intelligent supervision of granular data to generate comprehensive, advanced, high-quality dashboards that can utilize enterprise-level data models hosted in the cloud. As will be discussed in more detail below, embodiments can create intelligent data marts that store the most relevant information at the desired granularity level and are organized at different aggregation levels. Data from multiple sources can be integrated into cloud-independent and data format-independent processes, allowing the system to create new indicators and establish sets of relationships that support the construction of tabular semantic data models that power visualization tools. In one example, the proposed analysis framework uses highly optimized and automated partition loading techniques that involve segmenting data into smaller units. In addition, with the elimination of redundancy, custom data refresh techniques that selectively refresh only necessary partitions and intelligent data mart functions significantly reduce the required computing time. In some embodiments, the system can strategically ingest big data using a customized enterprise data model that intelligently segments data into smaller units and implements predetermined customized automatic refreshes without affecting data granularity and data quality. This approach has been proven to provide faster processing speeds and provide secure data integration. In addition, this type of granular data analysis enables enterprises to perform tooltip analysis without relying on coding expertise, thereby providing effective and efficient self-service tools to end users. The turnaround time for required insights is significantly reduced, which improves the usability of dashboards and provides end users with insights from the latest data, facilitating timely and informed decisions. In addition, some embodiments may incorporate generative artificial intelligence (AI) techniques when generating dashboards to additionally reduce (compress) backend data size and processing complexity.

[0025] For introductory purposes, Figure 1 A non-limiting example of an implementation of the proposed data analysis visualization system ("system") is depicted in . In various embodiments, a customer may desire that data stored in its cloud-based systems (such as a cloud-based extraction, transformation, and loading (ETL) and data integration services enterprise ("data integration enterprise")) be utilized and prepared for analysis using the proposed embodiments. Figure 1 In the present invention, data can be moved and refreshed along an automated pipeline from the data integration enterprise 100 to a data modeler, such as a platform as a service (PaaS) that provides enterprise-level data models in the cloud. Therefore, the proposed system can be integrated into an existing data integration enterprise 100, such as but not limited to Azure Data AWS IBM InfoSphere Matillion Apache Talend Big Data Pentaho Data SnapLogic IntelligentIntegration Platform MuleSoft Anypoint wait.

[0026] For example, when data is received into the data integration enterprise 100, the system performs an initial assessment of the size and velocity of the data in phase A. The first module 110 performs data transformation and aggregation on the received data and produces a first output including a final design of the data mart in phase B. The first module 110 may include, for example, tools for building, deploying, sharing, and maintaining scaled enterprise-level data solutions, such as, but not limited to, MongoDB Oracle Amazon Amazon DataStax Redis Enterprise CDP Data Couchbase wait.

[0027] In some embodiments, the final design is output as one or more CSV files. The second module 120 for data storage receives and stores the first output and also initiates a synchronized partition creation process in the table model to produce a second output shared with the customer's PaaS 160 (stage C). In various embodiments, the second module 120 may include a cloud-based repository for structured and unstructured data, such as, but not limited to, Azure Data Lake Storage Oracle Amazon CDP Data Databricks Lakehouse EDB Postgres Advanced wait.

[0028] The PaaS may also include a third module 130 and a fourth module 140. In this case, the second output is received by the third module 130, which uses the data to develop a model of the PaaS and may deploy the model to the cloud. In various embodiments, the third module 130 may include a development environment or code editor tool that can be used to create websites, web applications, web services, and mobile applications, such as, but not limited to, Microsoft Visual C++. Android IntelliJ wait.

[0029] In addition, at stage D, a partition intelligent refresh process using the data orchestrator is performed on the data, generating a third output shared with the fourth module 140. In various embodiments, the fourth module 140 may include a software application for configuring, managing, and operating a SQL server instance, such as, but not limited to, Microsoft SQL Server Management Studio. MySQL Teradata Oracle AmazonRelational Database Service SAP HANA Toad For Etc. In some embodiments, smart refresh is based on the refresh frequency requested by the customer. Therefore, during the initial design of the system, the customer may indicate the frequency or time interval at which data refresh and partition updates should occur, and the preference will be implemented into the system to allow automatic refresh to occur to match the customer's preferences. In other cases, customers may indicate their preferences for a first refresh rate for one type of data report (e.g., weekly refresh) and a different refresh rate for another type of data report (e.g., monthly refresh), and the system may create two different types of data tables in smart data that work at different refresh cycle rates. In the case where a partition is determined by the system to be too heavy or too large at the customer's expected refresh rate (based on an internal partition scoring mechanism), the system may automatically convert the refresh rate to a lower level (a more frequent refresh rate) to ensure that the partition score allows data analysis to occur without delay. Therefore, in some embodiments, the fourth module 140 can insert partitions, refresh tables, and be able to access data for a selected platform for self-service and enterprise business intelligence (BI) and data visualization, shown here as the fifth module 150, which can be used to generate key insights based on data. In various embodiments, the fifth module 150 may include tools for collating, managing, and analyzing data from various sources, such as Tableau Zoho and IMP Cognos wait.

[0030] Now go to Figure 2 , illustrates an embodiment of the architecture of the proposed data analysis system 200 to provide context to the reader. The architecture 200 can be understood as providing a simplified flow through the high-level key components of the system. Initially, customer data 210 to be analyzed is received, and a first decision 220 is made for the customer 210 to determine whether the recommended system should be implemented to perform the analysis. If the data does not require real-time analysis and is not computationally intensive (big data), traditional data loading techniques can be used to generate a visualization dashboard 270 that adequately covers the customer's needs. However, if the customer requires real-time big data analysis, the customer data 210 can be moved to the first stage of the system, where the smart data mart module 230 begins processing the customer data 210 and produces the final partitions based on CSV. The operation of the smart data mart module 230 will be referenced below. Figure 3 , Figure 4 and Figure 5 Discussed separately.

[0031] The output of the intelligent data mart module 230 is then transmitted to the synchronous partition creation module 240, which applies the optimized partitioning strategy to the data using logical sequence partitioning. In general, partitioning divides a table into multiple logical parts. The working principle of partitioning is to divide the table into multiple logical partition objects. Separate partitions, each containing a unique data segment, can then be processed incrementally in sequence or in parallel independently of other partitions, or completely excluded from the processing operation. Therefore, partitioning divides the data part that needs to be processed (refreshed) frequently from the data part that is processed less frequently. In other words, each partition can be processed (refreshed) independently of other partitions. In many cases, such as for tabular data, splitting a single partition of a table into multiple partitions can better utilize available resources for processing. As will be described in this article, the proposed system implements an effective model design and processing strategy that utilizes partitioning to eliminate unnecessary processor load and memory consumption while ensuring that data is refreshed frequently enough to reflect the latest data from the data source. The partitions defined for the model during model creation are repeated in the deployed model.

[0032] In some embodiments, the partition data can be transmitted to the refresh partition module 250, which uses data orchestration to refresh each partition. The refreshed partitions can be provided to the scheduled refresh module 260, which ensures that the partitions are maintained at their specified refresh rates (based on customer requests). Therefore, the scheduled refresh module 260 can automatically refresh the partitions according to the specified schedule of each data stream. The latest data is then used by the system to generate the desired visualization dashboard 270. The visualization dashboard 270 can be updated based on synchronized partitions, which reduces the amount of data that needs to be searched through intelligent filtering, which only searches for data units (partitions) related to the desired insights. In addition, refreshes can also be more efficient because only relevant partitions need to be refreshed from a larger data set, thereby reducing processing time. In different embodiments, the visualization dashboard 270 provides end users with the ability to create self-service-enabled visual effects based on granular data using unique prompt commands, in some cases by combining generative AI to create customized visual presentations.

[0033] It should be understood that the creation and availability of the system's intelligent data marts enables unique functionality to automatically partition big data into multidimensional data units at multiple levels. Therefore, the intelligent data mart supports seamless data connectors from the data warehouse layer to the visualization layer. In some embodiments, the system utilizes the creation of a table model on the back end by building a logical sequence of partitions, which in turn helps to facilitate synchronous scheduling of refresh events at the partition level. This type of approach ensures that when creating user queries through the front end, the system does not need to reference large data tables, but instead breaks down the data into smaller units (through innovative partitions in the intelligent data mart) on the back end, thereby achieving a bufferless retrieval process and significantly speeding up the response time of visualization generation.

[0034] Reference now Figure 3 , Figure 4 and Figure 5 , provides additional details about the characteristics of the analysis performed by the intelligent data mart module 230. Figure 3 , an example integration of the smart data mart module 230 with a customer's existing data ecosystem ("existing data") 300 is shown. In various embodiments, the existing data 300 may include a customer's existing data semantic layer. In one example, such existing data 300 may reside in a legacy system or cloud data lake with a conventional consistent refresh mechanism, and with or without a data grid architecture. It should be understood that the proposed data analysis system does not require any specific format or template for the customer's ecosystem, as it can easily connect or integrate across a wide variety of existing semantic data sources (e.g., the system is format agnostic).

[0035] In some embodiments, the existing data 300 may include a semantic layer having multiple dimensions and tables that can be fed into a standard data mart. In general, a data mart refers to a subject-oriented database that is usually a partitioned portion of an enterprise data warehouse. The subset of data stored in a data mart is typically associated with a specific unit, such as manufacturing, sales, etc. Data marts can speed up processes by allowing access to relevant information in a data warehouse in a shorter time, thereby forming a cost-effective way to quickly gain actionable insights. Transferring data from (multiple) source databases to one or more destination database(s) / data warehouse(s) may involve an extract, transform, and load (ETL) process that is responsible for each movement and processing of data to load the data into the destination database / data warehouse in an integrated form.

[0036] In one example, existing data 300 may be received into an initial processing ecosystem 390 of a larger data analytics system 200. In a first stage 310, a machine learning algorithm facilitates the collection of additional customer inputs, which will be used in part to determine how the data should be organized. In some embodiments, customers may describe how often they expect to see refreshed (updated) data, such as once a month, once a week, once a day, etc. In one embodiment, customers may select a refresh rate for each level of data. Thus, customer input will be used to determine desired levels of granularity such as recent details: monthly, weekly, daily, hourly, etc., as well as other features desired in reports.

[0037] The existing data 300 and relevant customer input are then passed to two separate modules, including a partitioner module 330 and a master tagging module 340, which can automatically prepare the data for building a smart data mart. In one example, the partitioner module 330 and the master tagging module 340 serve as tools to ingest customer data and generate an overview of various features or items in the data, such as counts of unique values ​​and / or multiple columns (e.g., Microsoft distinctcount in , etc.), and primary key analysis to determine one or more columns in the table that meet the conditions and are suitable as the primary key of the table, and to detail possible partitioning options.

[0038] In some embodiments, the partitioner module 330 includes an algorithm that is configured to propose one or more segmentation strategies across multiple data hierarchies based on customer input and the accuracy of the data. From these proposed strategies, the customer can select a strategy that suits their needs, or the selection can be made automatically based on preferences indicated in the customer input. Figure 4 , discusses additional details regarding the operation of the partitioner module 330. In general, the customer data 410 (e.g., Figure 3Existing data 300) can be received by partitioner module 330 and transferred along two paths based on the type of table involved. Thus, fact-based data 420 can be routed along a first path, while DIM-based data 430 can be routed along a second path.

[0039] In general, fact-based data 420 includes fact tables, and DIM-based data 430 includes dimension tables. For readers, these two can be understood as key components of the schema in the data warehouse. The fact table contains records that combine the attributes of different dimension tables. These records allow users to analyze different aspects of their business, which helps make decisions and improve the business. The dimension table provides context and context information for the measurements recorded in the fact table. Therefore, the difference between the fact table and the dimension table is that the dimension table contains the attributes on which the measurements in the fact table are based. In addition, when multiple fact tables are used, they can be organized using the "fact constellation mode". The fact table typically has two types of columns: columns containing facts and columns used as foreign keys linked to the dimension table. The primary key of the fact table is usually a composite key composed of all foreign keys in the table. Each fact table has a key, which is composed of the primary keys of all dimension tables connected to the fact table. The concatenation key is such a key that specifically identifies the fact table row. The fact table can save various types of metrics, such as additive metrics, non-additive metrics, and partially additive metrics, and stores important information in the data warehouse. Fact tables are therefore very useful for evaluating dimensional attributes because they provide additional value that can be used as independent variables. The granularity of a fact table is a common way to describe its layers. The granularity of a fact table refers to the most basic level at which a fact can be qualified. For example, the granularity of a sales fact table might be "number of byproducts sold per day by store," meaning that each entry in the table is uniquely identified by date, product, and store. In contrast, the term "dimension table" refers to a set of data related to any quantifiable event and is the basis of dimensional modeling. A dimension table has a column that acts as a primary key, allowing each dimension row or record to be uniquely identified. This key is used to link the dimension table to the fact table. Surrogate keys are system-generated keys that are typically used to uniquely identify rows in a dimension table.

[0040] In various embodiments, only fact-based data 420 is used to generate weighted scores for purposes of partitioner module 330. Figure 4As shown, fact-based data 420 is ingested by a scoring submodule 490 along with a specified preference 470 (e.g., a list of potential columns for partitioning and a weight for latency) . In the first stage 440, a first size score is calculated. If a column list is provided, the size score is calculated by checking the CSV file size of all combinations. If a column list is not provided, the first size score is calculated by checking the combination in the classification column. In the second stage 450, the second size score is calculated by checking the number of digital columns covered by each partition in the set of partitions created by force, and then calculating the minimum, maximum, and standard deviation of the aggregation. These scores will be used together to generate a weighted partitioner score 460, which can be used to evaluate the partitioning strategy. In other words, the partitioner module 330 can determine all possible combinations of primary keywords, sorting columns, etc. And based on these combinations, a sequence set of data of different lengths of all different combinations is calculated, and a score is assigned to each sequence set. The customer chooses whether to optimize in terms of storage size or in terms of synchronous loading based on the combination of the highest-scoring columns.

[0041] For clarity, Figure 5 A non-limiting example of strategy options that may be generated via the partitioner module 330 is depicted in FIG. As described above, the partitioner module 330 may receive customer data, such as an input data overview 510. In this case, the input data overview 510 includes two tables describing a larger data set. After an initial screening 520 involving data diagnostic techniques, the system may identify all feasible partitions. In this example, the shaded boxes represent data categories that are identified as the most feasible or optimal partitions based on the level of detail required by the customer. The screened data is then scored (e.g., see Figure 4 ). The scores are applied to the data to generate different multi-level partitions, each of which represents a strategy that seeks to optimize the number of partitions and the data in each partition.

[0042] In this example, the partitioning strategy 560 shows a first recommendation that includes a first option for partitioning at the store level and a second option for partitioning at the day level. In the second recommendation, the first option is to partition at the subcategory level and the second option is to partition at the day level. In different embodiments, additional levels may be included based on customer preferences. In addition, in some embodiments, rejected options 570 may also be presented, which are considered infeasible for the provided settings or cloud structure based on the calculated scores. It should be understood that the final scoring and strategy options do not require human intervention. In one embodiment, the partitioning strategy option with the highest score can then be identified as the top strategy that is most likely to provide the user with the desired data granularity while maintaining efficiency. Utilizing these optimized partitioning strategy options based on customer needs, the system can more easily develop and build smart data marts.

[0043] As before Figure 3 As described in, along with the partitioner module 330, the existing data is also processed by the master tag module 340. In general, the master tag module 340 implements an algorithm that generates a score based on multiple options for master tags suitable for various levels, from which the user can select an algorithm and continue, or the system can automatically select the highest score and continue. In different embodiments, the master tag module 340 can analyze the existing data and identify the customer's master tag (or primary keyword). Simply for the purpose of illustration, an example of a process that the master tag module 340 can use to aggregate data is shown below. In this example, a set of first data represented by Table 1 is received for store visits of two customers (a1 and a2) on different dates and their consumption history.

[0044] client date shop Consumption a1 4 / 4 / 2023 s1 26 a1 3 / 17 / 2023 s1 3 a1 3 / 9 / 2023 s4 99 a2 4 / 3 / 2023 s2 24 a2 2 / 27 / 2023 s2 96 a2 3 / 26 / 2023 s5 29

[0045] Table 1

[0046] The master tag module 340 can apply master tags based on the customer's store and output a new table (Table 2) that lists the same information, but in this case the store data has been changed to identify only the primary store for each customer (S1 for customer a1 and S2 for customer a2).

[0047]

[0048]

[0049] Table 2

[0050] This table data may then be summarized by the master tagging module 340 by reference to the master store to generate a third table (Table 3) that allows for a significant reduction in data size (in this case, approximately 40-50%).

[0051] client Main Store Consumption a1 S1 128 a2 S2 149

[0052] Table 3

[0053] Therefore, in different embodiments, the main tag feature of the system can flexibly perform fast aggregation at the visualization layer. The intelligent data mart constructed then is based on the output of the partitioner module 330 and the main tag module 340, and can integrate data from a wide range of big data sources (e.g., rows in billions and trillions) of different scopes to achieve real-time insights at the level of granularity (e.g., customers, stores, zip codes, etc.). The developed intelligent data mart also supports logical sorting of partitions and synchronous refresh of dynamic data, can fully support dashboards or reporting layers, and data is distributed and classified across more levels. In addition, the intelligent data mart can be used to generate different types of reports at different levels of granularity, while also supporting unit service functionality and ad hoc analysis. Therefore, if a customer expects to redesign the parameters of their report, rather than returning to the original data, they can directly access the intelligent data mart and run simple queries to retrieve the required insights, because the intelligent data mart acts as a single source of truth that greatly simplifies all queries. The intelligent data mart can maintain and express the customer's pre-existing data, and continue to perform refreshes to ensure that the data is up to date. To balance the need for frequent refreshes with processing efficiency, the system can perform refreshes based on data that has been aggregated using partitioners and master tags, so that the amount of data processed is reduced while most insights like all key performance indicators (KPIs) remain intact and accurate. In some embodiments, automatic data refresh using the proposed system will only refresh (rewrite and archive) specific partitions that have become outdated and replace it with new partition data (rather than inefficiently refreshing all partitions every time new incremental data is loaded to the backend).

[0054] In various embodiments, after the intelligent data mart is built, the data analysis system can automatically start the partitioning process based on the selected partitioning strategy, thereby loading customer data into the partitions. Figure 2 As introduced in , the data analysis system 200 can use the synchronous partition creation module 240 and the refresh partition module 250 to create and maintain the logical sequence of partitions as a table model. In order to compare and better illustrate the effectiveness of the proposed system, two tables are provided below. In Table 4, some characteristics of the conventional manual partition creation method for the data set are identified, while in Table 5, the characteristics of the synchronous partition creation method of the proposed system for the same data set are shown.

[0055]

[0056] Table 4

[0057]

[0058]

[0059] Table 5

[0060] Therefore, it should be appreciated that the back-end using a synchronous partition creation framework based on table modeling of an intelligent data mart can represent a far more efficient and streamlined process than conventional designs. Furthermore, since all partitions are loaded synchronously, the proposed system is independent of the actual number of partitions selected, thereby allowing efficient and intelligent management of a large number of partitions at multiple levels. In various embodiments, the data analysis system can then schedule refresh module 260 (in Figure 2 ) performs scheduled smart partition refreshes to ensure that the data in each partition is up to date. Figure 6 A simplified example of such a method for updating partitions 600 is depicted in . In this example, it should be understood that the partitions are arranged in a logical order based on the results of earlier modules and are closely associated with the table model.

[0061] In the first operation 610, an initial column is shown, which represents the data partitioned by date (partitioned by month, from February 2022 to January 2023). Below this column is a smaller column reflecting the updated data that needs to be merged into the existing data. For reference, these partitions will be marked as M1, M2, M3, ... Mn, and the data therein will be marked as {M1}, {M2}, {M3}, ... {Mn}. In the second operation 620, the system performs the first refresh of the partition. For simplicity, only 13 entries (M1-M13) are shown, but in other embodiments, the number of the list can be much larger. In the first refresh, the data increment (M13) is appended to the existing list. However, the system will not only add new data, but also replace the old partition to save memory. In this case, there is a new partition M13 of the data {M13} brought into the set, and the system will use it to automatically replace the earliest previous data. Therefore, by the second refresh of the third operation 630, the data {M13} has been inserted and overwrites the original {M1} data, so that M1 now contains {M13}. In addition, the old data {M1} is deleted and archived (represented by the independent M1 cell below the column), thereby optimizing storage, reducing costs and reducing data leakage. In addition, the archiving process ensures that there is a data backup that adapts to system failures (i.e., maintaining a second historical version) and can be used to recover the model if necessary. In some embodiments, the system's algorithm manages the history and decides which partition needs to be updated

[0062] In other words, an exchange can occur where the new data content replaces the old data content, but only within that particular partition. At the same time, the next updated data set (M14) can be moved into the loop. This approach accommodates the nature of the immutable partition names and number of partitions, which cannot be modified once the table model is deployed. In a fourth operation 640, the system moves from the second refresh to the third refresh. In this example, the current M1 data (i.e., now consisting of {M13}) remains intact and is considered to be up to date. In contrast, the new data of M14, which was appended in the previous stage, is now interchanged with the data held by M2. In other words, {M2} is replaced by {M14}. At the same time, the next updated data set (M15) can be moved into the loop, while the original data of M2 is archived.

[0063] In the fifth operation 650, a fourth refresh is performed. In this case, the current M2 data (i.e., now consisting of {M14}) remains intact and is considered to be up to date. Instead, the new data of M15, which was appended in the previous stage, is now interchanged with the data held by M3. In other words, {M3} is replaced by {M15}. At the same time, the next updated data set (M16) can be moved into the loop, and the original data of M3 is archived. The process is summarized in the sixth operation 660, which presents substantially the same columns from the first operation 610, but the contents of the first three partitions have been updated, and the original data of these three partitions have been safely archived.

[0064] Although for the sake of clarity for the reader, Figure 6The operations in are presented sequentially, but it should be understood that in various embodiments, these operations may occur in a substantially synchronous manner ("synchronous refresh") in the context of the system itself. This approach is in stark contrast to conventional data refresh mechanisms that are more time-consuming and prone to human error, which rely on a step-by-step sequence to move data from the data warehouse to the visualization layer through a series of steps that write and partition the data at manually defined intervals. The proposed embodiment incorporates synchronous refresh, such as by automatically moving data between layers (and at each level of partitioning) (each time a refresh is required) and automatically adding data when new data arrives, enabling a faster, error-free execution process to refresh large amounts of data. In other words, the proposed system employs a partition refresh technique that significantly reduces the computation time required to maintain up-to-date resources at a granular level. In one example, when the system moves new data from the data warehouse layer to the visualization layer at each refresh cycle (e.g., each new week requested by the user-selected partitioning strategy) through an intermediate table model tool, the system can replicate the individual partitions (e.g., the 104th partition layer) created and maintained in the corresponding smart data mart on the table model. The system's efficient synchronous refresh tool can automate the entire process and move data across all partition layers (e.g., 103 partition layers) to add new data to partition layer 104. Such automated methods - which reduce error rates - are conventionally not feasible in big data.

[0065] As previously mentioned, in various embodiments, the proposed system can employ generative AI techniques to generate visualizations of aggregated data stored in smart data marts. In some embodiments, generative AI is configured to filter data based on queries and automatically generate insights related to the queries. Thus, by providing generative AI with smart data marts as its source (where information is highly organized), visualizations can be quickly developed to provide insights that represent the latest available information. For example, in Figure 7 In the first operation 710, a user can access the front end of the system via a data analysis dashboard and submit a request to understand a specific aspect / feature of the data. In a second operation 720, the system calls a generative AI, which in response - in a third operation 730 - queries the smart data mart. Based on the available data, various visualizations can then be automatically generated.

[0066] In some embodiments, the generative AI may include a transformer-based deep learning neural network architecture. For example, a domain-specific language model may include an autoregressive language model (e.g., a generative pre-trained transformer (GPT) language prediction model) that uses deep learning to generate text in response to a query (e.g., generating answers to questions and / or generating replies to comments, etc.). In some embodiments, the visualization may be generated by passing data to a machine learning model, such as a generative pre-trained transformer (GPT), which is an open source artificial intelligence created by OpenAI (see Alec Radford et al.'s posting to https: / / OpenAI.com / blog (2023), including language models that are unsupervised multi-task learners, the entire contents of which are incorporated by reference). In some embodiments, the machine learning model may be trained on a dataset, such as the paraphrase antonyms from the word scrambling (PAWS) dataset.

[0067] Figure 8 800 is a schematic diagram of an environment 800 of an intelligent data analysis system 814 (or system 814) according to one embodiment. The environment 800 may include multiple components capable of performing the disclosed method. For example, the environment 800 includes a user device 804, a computing / server system 808, a data visualization platform 826, and a database 890 that may include an intelligent data mart. The components of the environment 800 can communicate with each other via a network 802. For example, the user device 804 can retrieve information from the database 890 via the network 802. In some embodiments, the network 802 can be a wide area network ("WAN"), such as the Internet. In other embodiments, the network 806 can be a local area network ("LAN").

[0068] like Figure 8 As shown, the components of system 814 can be installed in a computing system 808, which can have a memory 812 and a processor 810. Processor 810 can include a single device processor located on a single device, or it can include multiple device processors located on one or more physical devices. Memory 812 can include any type of storage, which can be physically located on one physical device, or on multiple physical devices. In some cases, computing system 808 can include one or more servers for hosting the system.

[0069] Although Figure 8One user device is shown, but it should be understood that one or more user devices may be used. For example, in some embodiments, the system may include two or three user devices. In some embodiments, the user device may be a computing device used by a user. For example, user device 804 may include a smartphone or a tablet computer. In other examples, user device 804 may include a laptop computer, a desktop computer, and / or other types of computing devices. User devices may be used to input, process, and display information. Reference Figure 8 , the environment 800 may also include a database 890 that stores processed partition data. The data may be retrieved by other components of the system 814. As described above, the system 814 may include a partitioner module 818, a master tag module 820, a partition module 822, and a synchronization refresh module 824. Each of these modules / components may be used to perform the operations described herein.

[0070] Fig. 9 is a flow chart illustrating an embodiment of a method 900 for processing and partitioning data based on a cloud computing infrastructure. The method 900 includes a first step 910 of receiving a first customer criterion and a first customer data set including tabular data at a partitioner module of a data analysis system, and a second step 920 of calculating a first size score for the tabular data at the partitioner module, the first size score identifying a first data increment that is most likely to meet the first customer criterion. In addition, a third step 930 includes generating a first partitioning strategy based on the first size score at the partitioner module, and a fourth step 940 includes implementing the first partitioning strategy at the data analysis system to produce a set of aggregated data in the tabular data, the sets including the first partition. A fifth step 950 of the method 900 includes storing the set data as part of an intelligent data mart in the data analysis system. A sixth step 960 includes receiving a first user query via a first user interface associated with a data visualization platform, and a seventh step 970 includes determining that the first user query is related to the first partition. Furthermore, method 900 includes an eighth step 980 of retrieving the first partition from the smart data mart via the data visualization platform, and a ninth step 990 of generating a first visualization based on the first partition for the data visualization platform and via a generative artificial intelligence (AI) module.

[0071] In other embodiments, the method may include additional steps or aspects. In one example, implementing the first partitioning strategy also includes creating multiple partitions in the tabular data in a substantially synchronized process. In some embodiments, the method also includes the following steps: receiving a second customer standard indicating a first refresh rate for the first type of data; determining, by reference to the smart data mart, that only a first portion of the first partition includes the first type of data; and refreshing only the first portion of the first partition at the first refresh rate based on the second customer standard. In another embodiment, the method also includes the following steps: receiving a third customer standard indicating a second refresh rate for the second type of data that is different from the first refresh rate; determining, by reference to the smart data mart, that only a second portion of the first partition includes the second type of data; and refreshing only the second portion of the first partition at the second refresh rate based on the third customer standard.

[0072] In different embodiments, additional steps may include calculating a second size score for the tabular data at a partitioner module, the second size score identifying a second data increment that is also likely to meet the first customer criteria; generating, at the partitioner module, a second partitioning strategy based on the second size score; and presenting the first partitioning strategy as a first option and presenting the second partitioning strategy as a second option via a second user interface of the data analysis system. In one example, the method also includes receiving a selection of the first option via the second user interface, and wherein implementing the first partitioning strategy is responsive to receiving the selection of the first option. In another example, the method also includes calculating a second size score for the tabular data at the partitioner module, the second size score identifying a second data increment that is less likely to meet the first customer criteria than the first data increment; generating, at the partitioner module, a second partitioning strategy based on the second size score; and presenting the second partitioning strategy as a rejected option for review via a second user interface of the data analysis system.

[0073] For the purposes of this application, an "interface" may be understood as a mechanism for conveying content to an application user through a client application. In some examples, an interface may include a pop-up window, a control, a drivable interface, an interactive button / option, or other objects that may be shown to a user through a native application UI, and a mechanism inherent to a particular application for presenting content associated with these native controls. In addition, the term "actuation" or "actuation event" refers to an event (or a specific sequence of events) associated with a particular input or the use of an application via an interface, which may trigger a change in the display of an application. In addition, a "native control" refers to a mechanism for delivering content to an application user through a client application. For example, a native control may include an actuatable or selectable option or "button" that may be presented to a user via a native application UI, a touch screen access point, a menu item, or other objects that may be shown to a user through a native application UI, a fragment of a larger interface, and a mechanism native to a particular application for presenting content associated with these native controls. The term "asset" refers to content that may be presented in association with a native control in a native application. As some non-limiting examples, assets may include text in an actuatable pop-up window, audio associated with an interactive click of a button or other native application object, video associated with a user interface, or other such information presentation.

[0074] It should be understood that the text, images, and specific application features shown in the figures are for illustrative purposes only and in no way limit the manner in which the application communicates or receives information. Moreover, in other embodiments, one or more options or other fields and text may appear differently and / or may be displayed or generated anywhere else on the screen(s) associated with the client system, including separate from, adjacent to, or surrounding the user interface. In other words, the figures present only one possible layout of the interface and do not in any way limit the presentation arrangement of any disclosed feature.

[0075] Embodiments may include a non-transitory computer readable medium (CRM) storing software including instructions executable by one or more computers that, when such executed, may cause the one or more computers to perform the disclosed methods. A non-transitory CRM may refer to a CRM that stores data for a short period of time or while powered, such as a storage device or random access memory (RAM). For example, a non-transitory computer readable medium may include a storage component such as a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, and / or a magnetic tape.

[0076] To provide additional context, in some embodiments, some of the processes described herein may be understood to operate in a system architecture that may include multiple virtual local area network (VLAN) workstations located at different locations that communicate with a main data center through dedicated virtual servers, such as a network server for a user interface, an application server for data processing, a database for data storage, etc. In general, a virtual server is a virtual machine (VM) that executes on a hardware component (e.g., a server). In some examples, multiple virtual machines may be deployed on one or more servers.

[0077] In various embodiments, the system may be at least partially hosted in a cloud computing environment that provides ready-made scalability and security. The cloud computing environment may include, for example, an environment that hosts document processing management services. The cloud computing environment may provide computing, software, data access, storage, and other services that do not require the end user to be aware of the physical location and configuration of the system(s) and / or device(s) hosting the policy management services. For example, the cloud computing environment may include a set of computing resources (collectively, "computing resources" and individually, "computing resources"). It is contemplated that embodiments of the present disclosure may utilize an appropriate cloud provider (e.g., provided by Amazon Web Services). TM AWS provided by GoogleTM, GCP provided by Microsoft TM Azure, etc.) to implement it.

[0078] The above methods, devices and processes can be implemented in many different ways and in many different combinations of hardware and software. For example, all or part of the embodiments may be a circuit device including an instruction processor, such as a central processing unit (CPU), a microcontroller or a microprocessor; or implemented as an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a field programmable gate array (FPGA); or implemented as a circuit device including discrete logic or other circuit components, including analog circuit components, digital circuit components or both; or any combination thereof.

[0079] Although various embodiments of the present invention have been described, the description is intended to be illustrative rather than restrictive, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the present invention. Therefore, the present invention is not limited except in accordance with the appended claims and their equivalents. In addition, various modifications and changes may be made within the scope of the appended claims.

Claims

1. A computer-implemented method for processing and partitioning data for a cloud computing-based infrastructure, the method comprising: receiving, at a partitioner module for a data analysis system, a first customer criterion and a first customer data set including tabular data; calculating, at the partitioner module, a first size score for the tabular data, the first size score identifying a first data increment most likely to satisfy the first client criterion; At the partitioner module, generating a first partitioning strategy based on the first size score; implementing the first partitioning strategy at the data analysis system to produce a set of aggregated data in the tabular data, the set including a first partition; storing the set of aggregated data in the data analysis system as part of a smart data mart; receiving a first user query via a first user interface associated with a data visualization platform; determining that the first user query is related to the first partition; Retrieving the first partition from the smart data mart via the data visualization platform; as well as A first visualization is generated based on the first partition for the data visualization platform and via a generative artificial intelligence (AI) module.

2. The method of claim 1, wherein implementing the first partitioning strategy further comprises: A plurality of partitions are created in the tabular data in a substantially synchronous process.

3. The method according to claim 1, further comprising: receiving a second client standard indicating a first refresh rate for a first type of data; By referring to the intelligent data mart, determining that only a first portion of the first partition includes the first type of data; as well as Based on the second client criteria, only the first portion of the first partition is refreshed at the first refresh rate.

4. The method according to claim 3, further comprising: receiving a third client standard indicating a second refresh rate for a second type of data different from the first refresh rate; By referring to the intelligent data mart, determining that only the second part of the first partition includes the second type of data; as well as Based on the third customer criteria, only the second portion of the first partition is refreshed at the second refresh rate.

5. The method according to claim 1, further comprising: calculating, at the partitioner module, a second size score for the tabular data, the second size score identifying a second data increment that may also satisfy the first client criteria; generating, at the partitioner module, a second partitioning strategy based on the second size score; as well as The first partitioning strategy is presented as a first option and the second partitioning strategy is presented as a second option via a second user interface for the data analysis system.

6. The method according to claim 5, further comprising: A selection of the first option is received via the second user interface, and wherein implementing the first partitioning strategy is in response to receiving the selection of the first option.

7. The method according to claim 1, further comprising: calculating, at the partitioner module, a second size score for the tabular data, the second size score identifying a second data increment that is less likely to satisfy the first customer criteria than the first data increment; generating, at the partitioner module, a second partitioning strategy based on the second size score; as well as The second partitioning strategy is presented as a rejected option for review via a second user interface to the data analysis system.

8. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers, the instructions, when executed, causing the one or more computers to process and partition data for a cloud computing-based infrastructure by: receiving, at a partitioner module for a data analysis system, a first customer criterion and a first customer data set including tabular data; calculating, at the partitioner module, a first size score for the tabular data, the first size score identifying a first data increment most likely to satisfy the first client criterion; At the partitioner module, generating a first partitioning strategy based on the first size score; implementing the first partitioning strategy at the data analysis system to produce a set of aggregated data in the tabular data, the set including a first partition; storing the set of aggregated data in the data analysis system as part of a smart data mart; receiving a first user query via a first user interface associated with a data visualization platform; determining that the first user query is related to the first partition; Retrieving the first partition from the smart data mart via the data visualization platform; as well as A first visualization is generated based on the first partition for the data visualization platform and via a generative artificial intelligence (AI) module.

9. The non-transitory computer-readable medium of claim 8, wherein the instructions further cause the one or more computers to create a plurality of partitions in the tabular data in a substantially synchronous process.

10. The non-transitory computer-readable medium of claim 8, wherein the instructions further cause the one or more computers to: receiving a second client standard indicating a first refresh rate for the first type of data; By referring to the intelligent data mart, determining that only a first portion of the first partition includes the first type of data; and Based on the second client criteria, only the first portion of the first partition is refreshed at the first refresh rate.

11. The non-transitory computer readable medium of claim 10, wherein the instructions further cause the one or more computers to: receiving a third client standard indicating a second refresh rate for a second type of data different from the first refresh rate; By referring to the intelligent data mart, determining that only the second part of the first partition includes the second type of data; and Based on the third customer criteria, only the second portion of the first partition is refreshed at the second refresh rate.

12. The non-transitory computer-readable medium of claim 8, wherein the instructions further cause the one or more computers to: calculating, at the partitioner module, a second size score for the tabular data, the second size score identifying a second data increment that may also satisfy the first client criteria; generating, at the partitioner module, a second partitioning strategy based on the second size score; as well as The first partitioning strategy is presented as a first option and the second partitioning strategy is presented as a second option via a second user interface for the data analysis system.

13. The non-transitory computer-readable medium of claim 12, wherein the instructions further cause the one or more computers to receive, via the second user interface, a selection of the first option, and wherein implementing the first partitioning strategy is in response to receiving the selection of the first option.

14. The non-transitory computer-readable medium of claim 8, wherein the instructions further cause the one or more computers to: calculating, at the partitioner module, a second size score for the tabular data, the second size score identifying a second data increment that is less likely to satisfy the first customer criteria than the first data increment; generating, at the partitioner module, a second partitioning strategy based on the second size score; as well as The second partitioning strategy is presented as a rejected option for review via a second user interface to the data analysis system.

15. A system for processing and partitioning data for a cloud computing-based infrastructure, the infrastructure comprising one or more computers and one or more storage devices storing instructions, the instructions being operable when executed by the one or more computers to cause the one or more computers to: receiving, at a partitioner module for a data analysis system, a first customer criterion and a first customer data set including tabular data; calculating, at the partitioner module, a first size score for the tabular data, the first size score identifying a first data increment most likely to satisfy the first client criterion; At the partitioner module, generating a first partitioning strategy based on the first size score; implementing the first partitioning strategy at the data analysis system to produce a set of aggregated data in the tabular data, the set including a first partition; storing the set of aggregated data in the data analysis system as part of a smart data mart; receiving a first user query via a first user interface associated with a data visualization platform; determining that the first user query is related to the first partition; Retrieving the first partition from the smart data mart via the data visualization platform; as well as A first visualization is generated based on the first partition for the data visualization platform and via a generative artificial intelligence (AI) module.

16. The system of claim 15, wherein the instructions further cause the one or more computers to create a plurality of partitions in the tabular data in a substantially synchronous process.

17. The system of claim 15, wherein the instructions further cause the one or more computers to: receiving a second client standard indicating a first refresh rate for a first type of data; By referring to the intelligent data mart, determining that only a first portion of the first partition includes the first type of data; and Based on the second client criteria, only the first portion of the first partition is refreshed at the first refresh rate.

18. The system of claim 17, wherein the instructions further cause the one or more computers to: receiving a third client standard indicating a second refresh rate for a second type of data different from the first refresh rate; By referring to the intelligent data mart, determining that only the second part of the first partition includes the second type of data; and Based on the third customer criteria, only the second portion of the first partition is refreshed at the second refresh rate.

19. The system of claim 15, wherein the instructions further cause the one or more computers to: calculating, at the partitioner module, a second size score for the tabular data, the second size score identifying a second data increment that may also satisfy the first client criteria; generating, at the partitioner module, a second partitioning strategy based on the second size score; as well as The first partitioning strategy is presented as a first option and the second partitioning strategy is presented as a second option via a second user interface for the data analysis system.

20. The system of claim 19, wherein the instructions further cause the one or more computers to receive, via the second user interface, a selection of the first option, and wherein implementing the first partitioning strategy is in response to receiving the selection of the first option.

Images