Enhanced code-free ETL system for automated big data conversion and sharing

Abstract

A data processing system and method provides automated extraction, translation, load (ETL) operations. A server coupled with the processor executes the instructions and extracts data from various sources, such as cloud storage, external APIs and direct uploads. The system may include a stream mode processing unit to process large data files in manageable blocks, thereby improving efficiency and reducing memory load. It performs integrity checks to ensure data accuracy and consistency. The system configures and applies predefined and custom transitions, which is facilitated by user-friendly interface and API integration. Custom conversion logic is integrated into the process, allowing adaptive data operations to be performed. The converted data is then verified and formatted to be loaded into different destination systems. The ETL process is efficient, extensible and user-friendly, and thus suitable for wide data processing applications.

Description

Background Art

[0001] The data management landscape, particularly in extracting, transforming, and loading (ETL) big data, faces significant challenges in efficiency and scalability. Traditional ETL systems, such as Amazon Web Services (AWS), Apache Airflow, and Apache Beam, rely primarily on code-based operations. This approach requires a large number of programmers to manage data processes, making it unsuitable for scenarios such as reconciling thousands of subscriptions, which require simplicity and automation for management oversight.

[0002] Existing systems struggle to process large datasets in a timely and efficient manner. Data reconciliation, billing, and pricing processes, which often involve oversized Excel files and other formats, require significant manual intervention, resulting in extended operational durations, sometimes lasting weeks. This inefficiency is exacerbated when processing large tabular data, as these systems face limitations in processing files above a certain size without impacting performance.

[0003] Furthermore, the lack of automation in the traditional ETL process required a high degree of manual input, leading to scalability issues. As the company grew and the number of global contractors increased, the need for a more robust and scalable system became apparent. The current system also had weak auditability due to the involvement of multiple systems and operators and its reliance on email communications that lacked transparency and traceability.

[0004] These shortcomings highlight the need for an innovative solution that can efficiently handle large-scale data processing, ensure scalability, and reduce reliance on manual operations, thereby improving overall operational efficiency and transparency in big data management. Summary of the Invention

[0005] The embodiments described herein improve big data management by providing a versatile, efficient, and user-friendly ETL system. Systems and methods are provided to address the complexity, scalability, and auditability of manual operations in traditional ETL processes, thereby providing a comprehensive solution for modern big data needs. Specifically, the disclosed embodiments provide a novel codeless ETL (extract, transform, load) system that is configured to automate and simplify the process of big data management, particularly for reconciliation, billing, and pricing operations in a business environment. The system and method are particularly configured to enable managers to automate data processing and sharing with minimal human intervention, in addition to traditional programmers.

[0006] In some embodiments, systems and methods are provided for processing large data streams, including, for example, large tabular data, without the typical size limitations found in conventional systems. In some embodiments, the systems and methods are scalable to process files much larger than traditional limits (e.g., exceeding 1 GB), making the disclosed embodiments particularly well-suited for processing large amounts of subscription data in media suitable for data exchange (e.g., Microsoft Excel, etc.).

[0007] In some embodiments, the system and method can adapt to various data formats and can integrate with external CRM systems via APIs, thereby customizing data representation using adapters. This flexibility can be extended to the system's output capabilities, which are not limited to Excel but also include other formats such as CSV and JSON.

[0008] In some embodiments, no-code operation is enabled, where the system is configured with a simplified user interface, facilitating uploading data, configuring transformations, and sharing results without requiring programming skills.

[0009] In some embodiments, comprehensive data processing supports a wide variety of transformations, such as copying, renaming columns, mathematical operations, currency conversion using real-time exchange rates, and data population from various sources. Data processing can be implemented through built-in programming languages ​​and the ability to extend functionality through low-code modifications.

[0010] In some embodiments, automated data validation and auditing can be performed, for example, before conversion begins to verify the consistency of each data portion and record the conversion steps, thereby enhancing transparency and accountability.

[0011] In some embodiments, the system can be scaled and adapted for different partners, markets, and / or products. The system's modular design allows for cloning of data flows for use with other partners, enhancing operational flexibility. In some embodiments, a test environment can be provided so that users can test configurations on sample data before applying it to production data, ensuring reliability and accuracy in data processing. A notification system includes email and / or internal notifications to alert users when transformations require manual intervention, streamlining the process flow. In some embodiments, the system ensures data privacy by limiting visibility within an account and providing the option to publish results to specific partners based on defined agreements. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Depicted is a system for automating codeless ETL according to some embodiments.

[0013] Figure 2An example operating environment for an automated codeless ETL system is shown in accordance with some embodiments.

[0014] Figure 3 Depicted are data extraction components of a system for automated codeless ETL, according to some embodiments.

[0015] Figure 4 Depicted are data transformation components of a system for automated codeless ETL, according to some embodiments.

[0016] Figure 5 Depicted is an automated process for codeless ETL according to some embodiments.

[0017] Figure 6 Depicted is a data extraction process for a system for automated codeless ETL, according to some embodiments.

[0018] Figure 7 Depicted is a data transformation process for a system for automated codeless ETL according to some embodiments.

[0019] Figure 8 An exemplary computer system is depicted in accordance with some embodiments.

[0020] Figure 9 Depicted is a user interface (UI) for automated codeless ETL, according to some embodiments.

[0021] Figure 10 Depicted is a user interface (UI) for automated codeless ETL, according to some embodiments. DETAILED DESCRIPTION

[0022] Embodiments of the present invention may be implemented by hardware, firmware, software, or any combination thereof. Embodiments of the present invention may also be implemented as instructions stored on a machine-readable medium, readable and executable by one or more processors. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices, etc. In addition, firmware, software, routines, and instructions may be described herein as performing certain actions. However, it should be understood that such descriptions are for narrative convenience only, and that such actions actually result from a computing device, processor, controller, or other device executing the firmware, software, routines, instructions, etc.

[0023] It should be understood that the operations shown in the exemplary method are not exhaustive, and other operations may be performed before, after, or between any of the shown operations. In some embodiments of the present invention, operations may be performed in different orders and / or in different ways.

[0024] Figure 1 A system 100 for automating big data reconciliation, billing, and pricing processes is shown. The system, referred to as a codeless ETL (extract, transform, load) system 100, is configured to facilitate automated processing and publishing of data streams, particularly data streams in tabular formats such as Excel files. Figure 1 , a codeless ETL system 100 is provided to automate and simplify complex data operations. In a non-limiting example, the codeless ETL system 100 for processing big data may include a data extraction module 102 and a data transformation module 104.

[0025] In some embodiments, data extraction module 102 may include functionality to interface with various data sources, including cloud-based platforms and external APIs. For example, data extraction module 102 may be configured to extract data from an affiliated customer's CRM system, thereby providing flexibility in acquiring initial data. The module is capable of processing various file types, currently focusing on Excel files, and is configured to read files in streaming mode, which mitigates technical limitations on file size.

[0026] In some embodiments, stream processing enables processing of large data sets, particularly when processing large tabular data, i.e., the enhanced codeless ETL system. Those skilled in the art will appreciate that stream processing technology enables efficient reading and processing of data in manageable chunks, significantly reducing memory overhead and facilitating real-time data processing. By fully leveraging stream processing, the system can dynamically adapt to varying data sizes and complexities, ensuring optimal performance without compromising processing speed or accuracy. Stream processing facilitates operations that require a continuous flow of data, such as real-time data transformation and loading, and provides a scalable solution for codeless ETL approaches.

[0027] In one non-limiting example, data extraction module 102 can utilize the .NET Framework and implement the FileStream.Read method to perform streaming data extraction from large files. The FileStream.Read method enables the module to read data in segments, allowing for processing large data sets without having to load the entire file into memory. By employing this data stream-based approach, the system can maximize performance while minimizing memory consumption, thereby facilitating a more scalable and responsive data extraction process.

[0028] In some embodiments, the data conversion module 104 may include a set of tools and functions for processing the extracted data. The module enables users to configure a series of transformations and operations on the data, which may include mathematical operations, currency conversion using real-time exchange rates, and data population from various sources. The module supports a variety of out-of-the-box transformations, such as Manufacturer's Suggested Retail Price (MSRP) and Cost of Goods Sold (COGS) calculations, profit and margin calculations, and Value Added Tax (VAT) rate application. In addition, the data conversion module 104 can support custom transformations via one or more application programming interfaces (APIs) (e.g., via the CLOUDBLUE CONNECT transformation extension module API), allowing for further customization and flexibility.

[0029] To enhance the data extraction capabilities of the system, in some embodiments, the data extraction module 102 can be configured to interact with the data transformation module 104 by providing automated transformation of the queued data for transformation (e.g., after extraction and validation of structure and content). The interaction between the data extraction module 102 and the data transformation module 104 can be configured to ensure that the extracted data conforms to the transformation criteria set and interpreted by the data transformation module 104, after which the data transformation module 104 can apply the necessary operations without requiring human intervention.

[0030] In some embodiments, the codeless ETL system 100 may include a UI module 106 configured for ease of use, allowing users to upload data files, configure transformations, and monitor the entire data processing workflow. The UI module 106 may also be configured to facilitate sharing processed data with partners in an automated or semi-automated mode. Figure 9 and Figure 10 shown.

[0031] like Figure 9 As shown, UI 900 is used for an enhanced codeless ETL system designed for big data transformation and sharing, which can be implemented to allow a wide variety of transformation logic to be implemented, for example, through data transformation module 104, without requiring coding expertise. In a non-limiting example, a user can perform the pricing flow customization shown to implement fine-grained rules, such as product lookup rules, retrieval of suggested list prices (MSRPs) and cost of goods sold (COGS) values, etc., in a UI configured for ease of use. Figure 10A UI 1000 for an enhanced codeless ETL system designed for big data transformation and sharing is shown. As shown, UI 1000 includes an outbound pricing batch management section, including status indicators 1010 and 1012. Status indicator 1010 displays the overall progress of the batch processing, indicating how much of the batch processing has been completed as a percentage. Another status indicator 1012 displays the final or progress status of a single transformation by using a similar percentage completion metric. These indicators form a comprehensive dashboard that allows users to efficiently monitor and manage data transformation tasks, including viewing batch details, processing transformation requests, and tracking overall progress. Both UI 900 and UI 1000 are designed to streamline operations for users without coding expertise, enabling them to configure, execute, and monitor ETL processes through a simplified and intuitive interface.

[0032] The data conversion module 104 can be configured to initiate one or more preconfigured or custom conversions upon receiving data from the data extraction module 102. The data conversion module 104 can be configured to communicate with the UI module 106 to receive user-defined conversion rules and apply them accordingly. In some embodiments, a bidirectional information flow between the data conversion module 104 and the UI module 106 enables the user to set conversion parameters and receive feedback on the conversion results.

[0033] In some embodiments, the system 100 may further include an automated data validation and auditing module 108. This module is responsible for validating the data structure line by line according to a predefined sample file structure. It also provides functionality for checking cell values ​​based on user-defined constraints (e.g., allowing empty cells or decimal precision).

[0034] In some embodiments, system 100 may include a data publishing module 110. This allows processed data to be disseminated to various stakeholders, such as vendors, distributors, and dealers. This module is particularly adept at handling complex computation chains where data needs to go through multiple processing and approval stages before being shared.

[0035] The UI module 106 can provide a central hub for user interaction with the system. For example, it can collect input for the data transformation module 104 and display processed data from the data publishing module 110. Thus, the UI module 106 can integrate the transformation rules applied by the data transformation module 104 and the output prepared for publication, thereby providing an informative overview of the data lifecycle within the system 100.

[0036] In some embodiments, data publishing module 110 can receive processed and transformed data from data transformation module 104 and format the data into publishable content for distribution to intended users / entities. In terms of scalability and flexibility, codeless ETL system 100 can enable account administrators to clone data flows for use with different partners or marketplaces. This feature ensures that the system can adapt and scale to growing and changing business needs.

[0037] Additionally, the system includes a notification module 112 that alerts users via email and an internal notification system when manual intervention is required during the conversion process. This module helps streamline operations and ensures timely manual input when necessary. The data publishing module 110 can operatively interact with the notification module 112 to notify users of the publishing status and any required actions, thereby ensuring that all users and entities have access to current data.

[0038] The system's architecture allows data flows to be connected to partner accounts, marketplaces, and individual products or programs. This capability ensures that processed data is relevant and can be used effectively for its intended purpose, whether it's pricing, billing, or reconciliation.

[0039] In some embodiments, the system 100 may include a testing module 114 that enables administrators to test configurations on sample data before deploying transformations to production data. This module is provided to ensure that data processing workflows remain reliable and accurate. In some embodiments, the testing module 114 may be configured to provide a controlled environment for testing data transformations applied by the data transformation module 104.

[0040] In some embodiments, the notification module 112 can coordinate between the test module 114 and the UI module 106. The notification module 112 can alert the system administrator of any conversion errors detected during testing and prompt the UI module 106 to make necessary adjustments before publishing to ensure data integrity.

[0041] In some embodiments, system 100 may include an internal database and logging module 116 that maintains a comprehensive log of all transformation changes, including the time of action, the users involved, any relevant comments, etc. This module enhances system transparency and traceability, allowing users to effectively audit the transformation process. Testing module 114 can be configured to integrate with internal database and logging module 116 to record all test results for audit and compliance purposes. Feedback loops between modules ensure that only validated and accurate transformations are deployed to production.

[0042] In some embodiments, the data processing engine 118, part of the cloud platform's EaaS module, handles the actual data processing tasks. It uses a multi-threaded approach to break down source data into sections that can be processed in parallel, ensuring efficient and rapid processing of large data sets. The data processing engine 118, which performs data processing tasks, can be integrated with the data extraction module 102 and the data transformation module 104 to ensure efficient allocation of computing resources and optimize the data flow between the extraction and transformation stages to improve performance. Thus, the codeless ETL system 100 is configured to automate the processing, transformation, and sharing of large data sets for reconciliation, billing, and pricing processes within a business environment.

[0043] Figure 2 A block diagram of an operating environment 200 is shown in which a codeless ETL system 216 may be implemented. The codeless ETL system 216 may be the system described previously in Figure 1 . Operating environment 200 may include various components, each of which contributes to the efficient operation of the codeless ETL system. It is important to note that in other embodiments, operating environment 200 may not have all of the components listed and / or may have other elements in place of or in addition to the components listed here. In the present invention, users of the system may be synonymously referred to as customers, clients, or operators.

[0044] The operating environment 200 is the environment in which the codeless ETL system 216 exists and effectively operates. The user system 212 can be any machine or system that a user uses to interact with the codeless ETL system 216. For example, the user system 212 can include a handheld computing device, a mobile phone, a laptop computer, a workstation, or a network of computing devices. Figure 2 (and Figure 1 ), a user system 212 interacts with a codeless ETL system 216 via a network 214, which may include several components configured to perform various aspects of a codeless ETL process.

[0045] The codeless ETL system 216 includes an integrated data processing and transformation platform that is particularly well-suited for executing such automated processes, enabling complex operations to be performed by users with limited knowledge of the system's technical complexities. Instead, users can fully utilize the system to perform various data operations, such as extracting, transforming, and loading data. The application platform 218 provides the framework that enables the system's 216 applications to run, including hardware and software resources such as operating systems. The codeless ETL system 216 may include the application platform 218, which facilitates the creation, management, and execution of various applications developed by the system's provider, users, or third-party developers.

[0046] System 200 may include several processes for data manipulation and transformation. For example, it can identify and extract data from various sources, transform that data based on user-configured rules and processes, and load the transformed data into a specified destination. These functions are implemented by various components of the system, including codeless ETL data store 222, system data store 224, process space 228, and processor system 217.

[0047] The network 214 is any network or combination of networks of devices that communicate with each other. It can be a local area network, a wide area network, a telephone network, a wireless network, or other suitable configuration. Although the most common type of network in use today is a TCP / IP network, such as the Internet, it should be understood that the network 214 is not limited to this configuration and can include other network types.

[0048] User system 212 may communicate with system 216 using TCP / IP and, at a higher network level, other common Internet protocols such as HTTP, FTP, or WAP. In some implementations, user system 212 may include a "browser" for sending and receiving messages to and from system 216 over network 214. The interface between system 216 and network 214 may include load sharing functionality, such as a round-robin request distributor, to balance the load and evenly distribute incoming requests across multiple servers.

[0049] In one embodiment, Figure 2 The illustrated system 216 implements a codeless ETL platform. For example, the system 216 may include an application server configured to implement and execute a codeless ETL software application, provide relevant data, code, forms, web pages, and other information to and from the user system 212, and store and retrieve relevant data and objects in a database system.

[0050] Figure 2 Several elements in the illustrated system include conventional, well-known elements that are only briefly explained here. For example, each user system 212 may include a personal computer, a laptop computer, a personal digital assistant (PDA), a mobile phone, or any WAP-enabled device capable of directly or indirectly connecting to the Internet or other network. Each user system 212 typically runs a browser, thereby allowing the user to access, process, and view available information and applications from system 216 via network 214.

[0051] According to one embodiment, each user system 212 and all of its components are configurable to an operator using an application, such as a browser. Similarly, system 216 and all of its components are configurable to an operator using an application comprising computer code executed by a processor, such as processor system 217. Program code 228 may include instructions that can be used to program a computer to perform any of the processes of the embodiments described herein.

[0052] Figure 2 An arrangement of elements of system 216 is shown, including a network interface 220, an application platform 218, a codeless ETL data store 222 for storing configuration and transformation data, a system data store 224 for storing operational data, program code 228 for implementing various functions of system 216, and a process space 228 for executing service processes and system-specific processes. Process space 228 can be used to run applications and hosted services as part of the codeless ETL platform. In some embodiments, process space 228 can include a multi-threaded processing environment in which different data transformation tasks can be executed in parallel, thereby enhancing the overall efficiency of the system.

[0053] Codeless ETL data store 222 is a key component for the operation of system 216. It stores the configurations, rules, and templates used in the data transformation process. This store is designed to be highly flexible and scalable, adapting to the ever-changing needs of various data processing tasks. It enables users to easily and efficiently store and retrieve transformation configurations, thereby facilitating the rapid deployment of data processing tasks.

[0054] System data storage 224 provides a repository for operational data of system 216. This can include logs, user information, system status reports, and other data critical to the smooth operation and monitoring of the system. This storage is built to handle large amounts of data and is optimized for fast access and data retrieval, ensuring that system performance remains at an optimal level.

[0055] Program code 228 comprises the core software logic that drives the functionality of system 216. It may include algorithms and processing logic for extracting data from various sources, applying transformations based on user configurations, and loading the processed data into the desired destination. This code is provided to facilitate flexibility and customization, enabling users to define their data processing logic without requiring extensive programming knowledge.

[0056] Application platform 218 is the backbone of system 216, providing the necessary hardware and software infrastructure for the execution of program code 228 and for storing and retrieving data in and from codeless ETL data store 222 and system data store 224. The platform is configured to be robust and scalable, capable of supporting the processing of large amounts of data while maintaining high performance and reliability.

[0057] Processor system 217 is responsible for executing the instructions of program code 228. It includes one or more processor units capable of handling the computationally intensive tasks involved in extracting, transforming, and loading data. The processor system is selected and optimized to handle the specific requirements of the codeless ETL process, ensuring that data operations are performed quickly and efficiently.

[0058] Process space 228 is where the actual execution of data processing tasks occurs. It provides an isolated and secure environment for running the various processes and applications that form part of system 216. This space is configured to maximize processing efficiency and ensure the integrity and security of the processed data.

[0059] Figure 2Several elements in the illustrated system include conventional, well-known elements that are only briefly explained here. For example, each user system 212 can include a desktop personal computer, a workstation notebook computer, a PDA, a mobile phone or any device that supports a wireless access protocol (WAP) or any other computing device that can be directly or indirectly connected to the Internet or other networks. Each user system 212 usually runs an HTTP client, such as a browser, such as a browser of Microsoft's Edge browser, Google's Chrome browser, Opera, or a browser that supports WAP in the case of a mobile phone, PDA or other wireless device, thereby allowing the client of the user system 212 to access, process and view information, pages and applications available to it from system 216 through network 214. Each user system 212 can also usually include one or more user interface devices, such as a keyboard, mouse, trackball, touch pad, touch screen, pen etc., for interacting with a graphical user interface (GUI) provided by a browser on a display (e.g., a monitor screen, LCD display etc.) in conjunction with pages, forms, applications and other information provided by system 216 or other systems or servers. For example, the user interface device can be used to access data and applications hosted by the system 216 and perform searches on stored data, or otherwise allow a user to interact with various GUI pages that may be presented to the user. As described above, embodiments are suitable for use with the Internet, which refers to a particular global network of networks. However, it should be understood that other networks can be used in place of the Internet, such as an intranet, an extranet, a virtual private network (VPN), a non-TCP / IP based network, any local area network or wide area network, etc.

[0060] According to one embodiment, each user system 212 and all of its components are operator-configurable using an application, such as a browser, that includes computer code executed using a central processing unit, such as an Intel Pentium RTM processor. Similarly, the system and all of its components can be operator-configurable using an application that includes computer code executed using a central processing unit, such as a processor system 217, which can include an Intel Pentium RTM processor, etc., and / or multiple processor units. A computer program product embodiment can include a machine-readable storage medium (medium) having instructions stored thereon that can be used to program a computer to perform any of the processes of the embodiments described herein. For example, the computer code for operating and configuring the system 216 to interconnect and process the web pages, applications, and other data and media content described herein is downloaded and stored on a hard disk, but the entire program code or portions thereof may also be stored in any other well-known volatile or non-volatile memory medium or device, such as ROM or RAM, or provided on any medium capable of storing program code, such as any type of rotating media, including floppy disks, optical disks, digital versatile disks (DVDs), compact disks (CDs), microdrives and magneto-optical disks, as well as magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data. In addition, the entire program code or portions thereof may be transmitted and downloaded from a software source or from another server as is well known, via a transmission medium (e.g., via the Internet), or transmitted via any other well-known conventional network connection (e.g., an extranet, VPN, local area network, etc.) using any well-known communication medium and protocol (e.g., TCP / IP, HTTP, HTTPS, Ethernet, etc.). It should also be understood that the computer code used to implement the embodiments may be implemented in any programming language that can be executed on the client system and / or server or server system, such as C, C++, HTML, any other markup language, Java.TM., JavaScript, ActiveX, any other scripting language, such as VBScript, and many other well-known programming languages ​​may be used (Java.TM. is a trademark of Sun Microsystems, Inc.).

[0061] Thus, an operating environment 200 is provided for the codeless ETL system 100. It enables automated processing, transformation, and sharing of big data for various business applications. The operating environment is configured to be flexible, scalable, and user-friendly, thereby meeting the diverse needs of users involved in complex data operations.

[0062] Figure 3The data extraction module 300 is shown, which can be an embodiment of the data extraction module 102 within the codeless ETL system 100. This module is specifically configured to efficiently extract and process data from various sources. Figure 3 , a data extraction system 300 is provided to handle the initial stage of data processing. In a non-limiting example, the data extraction system 300 for streamlining data collection may include a stream mode processing unit 302 and a source compatible interface 304 .

[0063] Streaming mode processing unit 302 may include sophisticated algorithms and techniques for reading and processing data in streaming mode. This functionality allows processing large data files, such as massive Excel files, by processing the data in manageable chunks rather than loading the entire file into memory. This approach significantly improves the efficiency and performance of the data extraction process, especially when processing large data sets.

[0064] The streaming data processing capabilities of the data extraction module 300, as part of the codeless ETL system 100, provide a technical approach to addressing the challenges of efficiently processing large data files. The streaming processing unit 302 can be designed to process data files to efficiently perform batch processing. In some embodiments, the streaming processing unit 302 can be configured to perform real-time data reading and processing. In some embodiments, the streaming processing unit 302 can be configured to read and process large data files by breaking them into smaller, manageable chunks or batches. Thus, the data extraction module 300 can process data in a controlled, efficient manner suitable for processing large files.

[0065] A batch can be processed sequentially, ensuring that the system's memory usage remains within manageable limits. This approach significantly reduces the risk of memory overload, which often occurs when processing large files. By segmenting the data, the module can process each segment individually, applying the necessary extraction processes before moving on to the next segment. This batch processing approach also allows the system to pause, resume, or restart processing of data files as needed, adding a layer of flexibility and control.

[0066] The streaming mode processing unit 302 can employ advanced algorithms to efficiently parse and process each batch. These algorithms are configured to optimize the reading, extraction, and initial processing of data, ensuring that each batch is processed quickly and accurately. This allows data-intensive applications to be executed with optimized time and resources. Furthermore, the data extraction module 300 can adjust the size of each batch based on the overall file size and the current system load. This scalability ensures that the module can adapt to varying file sizes and system capacities, maintaining optimal performance regardless of data volume.

[0067] Streaming mode processing unit 302 implements efficient error handling and data integrity checking. Because the system processes data in continuous batches, validation and error checking routines can be performed independently on each batch. This granularity in processing data allows inconsistencies or errors in specific batches to be identified and resolved without affecting the entire file. Thus, streaming mode processing unit 302 processes data sequentially, processing a portion of the data at a time, so that data is processed in a controlled, batch-by-batch manner as it is read.

[0068] Therefore, the data ingestion module 300 can manage large data files using streaming processing. This approach provides a balance between efficiency and control, enabling the system to process large data files while reducing memory pressure and increasing processing speed. The module's algorithms are customized to optimize batch processing, ensuring scalability, accurate data manipulation, and robust error management. While different from real-time data streaming, this approach offers significant advantages in resource management and operational flexibility, particularly in scenarios involving large-scale data ingestion.

[0069] The source-compatible interface 304 is configured to enable the data extraction module 300 to interface with a variety of data sources. This can include cloud storage platforms, CRM systems, external APIs, and direct user uploads. The interface is equipped with a range of connectors and adapters, allowing it to adapt to diverse data environments and formats. This adaptability ensures that the module can effectively extract data regardless of its source or format, whether Excel, CSV, JSON, or other common data formats. The data extraction module 300 demonstrates the technical capability to interface with a wide range of data sources. This versatility is achieved through the integration of multiple data connectors and adapters. These components are designed to connect with CRM systems, leveraging their databases to efficiently extract relevant data. Furthermore, the module is equipped with interfaces for communicating with various external APIs, enabling it to access and retrieve data from a range of external systems and platforms. The module's adaptability extends to handling diverse file formats, including but not limited to Excel, CSV, and JSON. This functionality is supported by sophisticated parsing algorithms that can interpret and process the structural nuances of these diverse formats, ensuring accurate data extraction and minimizing format-related errors.

[0070] Furthermore, the data extraction module 300 incorporates data integrity mechanisms to ensure the accuracy and consistency of the extracted data. These mechanisms include a validation process that checks the structure and content of the data against predefined standards, identifying any anomalies or inconsistencies. This feature is crucial for maintaining the quality and reliability of the data as it passes through subsequent processing stages in the ETL system.

[0071] Advanced data integrity assurance mechanisms systematically evaluate data against predefined criteria, identifying discrepancies such as missing values, format inconsistencies, and data corruption to ensure the accuracy of each data batch and maintain consistency across the entire dataset. Therefore, the data extraction module 300 is configured to efficiently extract data from various sources while ensuring data integrity, laying a solid foundation for the effective operation of the system 100 in various data processing scenarios.

[0072] Figure 4 An embodiment of a data transformation module 400 is depicted, which may be a specific embodiment of the data transformation module 104 in the codeless ETL system 100. This module facilitates advanced data transformations required in various business applications. Figure 4 , the data conversion system 400 may include a conversion processing unit 402 and a custom conversion integration unit 404 , which are configured to process complex data operations.

[0073] The transformation processing unit 402 employs algorithms to perform a range of data transformations. These transformations include mathematical operations, currency conversions, and advanced data structuring. This unit uses programming languages ​​such as JQ, allowing for complex and flexible data manipulation. This capability enables users to customize data transformations based on specific needs.

[0074] The custom transformation integration unit 404 enables users to add their own custom transformations to the data transformation module 400. This unit integrates with the CloudBlue Connect transformation extension module API, providing a platform for incorporating user-defined transformations. Users write custom code according to the system's guidance, and the system integrates it, thus extending the module's functionality.

[0075] In some embodiments, the data conversion module 400 can be operably connected to an automation platform (e.g., CLOUDBLUE CONNECT), for example, via an API, to enable integration of complex distribution and supply chain applications. This integration facilitates interaction with various entities in the distribution chain, including vendors, distributors, and dealers. The module can integrate an API gateway to enable users to create custom transformations tailored to specific supply chain and distribution scenarios. This can include automating interactions between different entities or integrating various distribution chain data formats into a unified processing system.

[0076] Integration enables the inclusion of transformation logic tailored to specific user needs. Users can develop transformations that include unique business logic or data manipulation routines not available in the standard set of transformations. The system then incorporates user-defined transformations into its codeless operations framework. This capability is supported by a dynamic linking mechanism, allowing the system to recognize and execute these custom transformations as if they were native components of the module. This enhances the system's flexibility and enables it to adapt to business opportunities and evolving data processing requirements.

[0077] The data transformation module 400 also features a user-friendly configuration interface. This interface streamlines the setup and application of transformations, making the process accessible even to users with limited programming expertise. It enables straightforward selection, customization, and testing of transformations, ensuring accurate data processing according to user instructions. In line with the system's code-free operation principles, the interface provides users with a visual representation of data flows and transformation processes, enabling them to understand and configure transformations without having to write or understand complex code. The interface also includes features such as drag-and-drop capabilities, pre-built transformation templates, and interactive guides that help streamline the configuration process. This user-centric design approach enables a wider range of users, including those with minimal technical backgrounds, to fully utilize the advanced data processing capabilities, promoting more inclusive use of the system across a variety of organizational roles. Consequently, the data transformation module 400 is configured to handle a wide range of data transformations, from standard processes to customized ones, significantly enhancing the versatility of the system 100 in various data transformation contexts.

[0078] It should be understood that the operations shown in the exemplary method are not exhaustive, and other operations may be performed before, after, or between any of the operations shown. In some embodiments of the present invention, operations may be performed in different orders and / or in different ways.

[0079] Figure 5 is a data flow diagram of method 500, which is an embodiment of performing a codeless ETL process within system 100. This method provides an efficient approach to data manipulation and processing, including data extraction, transformation, and loading. Method 500 is configured to efficiently manage data workflows, ensuring accuracy, scalability, and adaptability to various data environments. Based on the disclosure herein, the operations in method 500 may be performed in a different order and / or in a different manner.

[0080] At operation 502, the computing device may perform data source identification. This step includes determining the source of the data, which may be a CRM system, an external API, cloud storage, or a direct upload. This process may include evaluating the expected data format and structure from these sources to ensure that subsequent extraction processes are tailored to efficiently process the data.

[0081] At operation 504, the computing device may initiate the data extraction process. This step involves the data extraction module 300 employing its streaming data processing technology. The module processes data in manageable batches, thereby reducing memory load and improving processing efficiency. This step is crucial for processing large data files, such as massive Excel files, where loading the entire file into memory is impractical.

[0082] At operation 506, the computing device may perform a data integrity check. This verification process may include carefully examining the data for structural accuracy, content consistency, and identifying any anomalies or discrepancies. The system employs a series of algorithms configured to detect and resolve issues such as missing values, format inconsistencies, and data corruption. This step is crucial to ensuring that the extracted data is reliable and suitable for further processing.

[0083] At operation 508, the computing device can determine whether the extraction and validation were successful. If successful, the system transitions to the data conversion phase. During this step, the user interacts with the user-friendly configuration interface of the data conversion module 400. The user can select from a range of pre-built conversion templates or configure a custom conversion. This process can include defining conversion logic, such as mathematical operations, currency conversion, and data structuring, tailored to the specific requirements of the data processing task.

[0084] At operation 510, the computing device can perform optional specialized data operations. If the user desires, the system provides the ability to integrate custom transformations. Users can write and integrate their transformation code via an API (e.g., the CLOUDBLUE CONNECT API). Operation 510 can include applying unique business logic or data manipulation routines, thereby enhancing the system's adaptability to specific user needs.

[0085] At operation 512, after setting the configuration, the computing device can perform data transformation. Operation 512 can include applying the defined transformation logic to the extracted data. The transformation processing unit 402 of the module 400 can be configured to perform operation 512 so that each data batch is processed according to the configured rules and conditions. Operation 512 can include converting the raw extracted data into a format that is meaningful and useful to the end user.

[0086] At operation 514, the computing device may perform loading of the processed data into a designated destination system. This may include a database, data warehouse, or other storage system. The loading process is configured to be efficient, thereby ensuring that the transformed data is accurately and completely integrated into the target system.

[0087] At operation 516, the computing device can continuously monitor and manage the data flow. This can include tracking the progress of data extraction, transformation, and loading, as well as identifying and resolving any issues that may arise during the process. The system provides users with tools and interfaces to oversee the ETL workflow, providing insight into each step and the ability to intervene when necessary.

[0088] At operation 518, the computing device can collect feedback and iterate the process. Based on the performance of the ETL workflow and user feedback, the system can adjust and refine its process. This may involve fine-tuning transformation configurations, optimizing extraction methods, or modifying data loading techniques. This iterative approach ensures continuous improvement of the ETL process, adapting to changing data requirements and user needs.

[0089] Thus, the method 500 provides a codeless ETL process performed by a computing device such as the system 100. The codeless ETL process manages complex data workflows, thereby implementing different data processing requirements in various business environments.

[0090] Figure 6 Illustrated is a method 600 for performing a data extraction process within the codeless ETL system 100. The method 600 ensures a reliable extraction phase of the ETL process.

[0091] At operation 602, the computing device performs data source identification and evaluation. This step involves detailed analysis of data sources, including CRM systems, external APIs, cloud storage platforms, or direct uploads. It evaluates the format, structure, and specific characteristics of the data from these sources to prepare for optimized extraction.

[0092] At operation 604, the computing device selects the most appropriate extraction method based on the nature of the data source. This may involve direct API calls to cloud-based sources, SQL queries to database systems, or parsing algorithms for file-based sources. The selection is based on efficiency and compatibility of the method with the data source.

[0093] At operation 606, the computing device configures the necessary data connectors and adapters. This step ensures integration with the data source, enabling the system to efficiently access and retrieve data. Connectors and adapters are customized to handle the specific data protocols and formats inherent in the source.

[0094] At operation 608, the computing device initiates an extraction process in which data is extracted from the source based on the configured methods and connectors. This process is performed while maintaining data integrity, thereby ensuring that data is accurately captured from the source without any loss or corruption.

[0095] Particularly for large data files, at operation 610, the computing device performs a streaming data processing method. This involves reading and processing data in manageable chunks, effectively reducing memory load and increasing processing speed. The system dynamically adjusts the size of these chunks based on file size and system capacity.

[0096] At operation 612, the computing device performs a series of data integrity checks and validation procedures. This may include verifying the data format, ensuring structural accuracy, and detecting any anomalies or inconsistencies. These checks are crucial to ensuring the quality and reliability of the extracted data.

[0097] At operation 614, the computing device checks whether errors or problems were encountered during the extraction. If so, the computing device invokes one or more mechanisms to handle and resolve these errors or problems. This may involve retrying the extraction process, adjusting extraction parameters, or flagging the problem for user intervention. Thus, operation 614 performs verification and correction to maintain the continuity and efficiency of the extraction process.

[0098] At operation 616, the computing device formats and standardizes the extracted data. This step ensures that the data conforms to a consistent structure and format, thereby facilitating integration with subsequent transformation processes in the ETL workflow.

[0099] After successful extraction and standardization, the computing device transitions to the data transformation phase at operation 618. The extracted data is now ready for further manipulation and processing as defined in subsequent phases of the ETL system. Thus, method 600 provides an efficient method for extracting data in a codeless ETL system.

[0100] Figure 7 Illustrated is a method 700 for performing a data transformation process within the codeless ETL system 100. The method 700 provides an efficient method for transforming extracted data, ensuring its relevance, accuracy, and suitability for its intended use.

[0101] At operation 702, the computing device begins to receive data from the data extraction module. The data arrives in raw format directly from various data sources and needs to be converted to obtain the required structure and content.

[0102] At operation 704, the computing device analyzes the structure of the received data and identifies transformation requirements. This step includes understanding the ultimate use of the data (which can range from analytical processing to report generation) and defining the transformation logic required to meet these goals.

[0103] At operation 706, the computing device displays pre-built transformation templates suitable for common data scenarios. In this step, the user selects an appropriate template that aligns with their data transformation goals. These templates simplify the transformation process, especially for users without extensive technical expertise.

[0104] In operation 708, the computing device optionally implements scenarios requiring specialized data manipulation, thereby allowing the user to configure custom transformations. This involves writing transformation logic or scripts (possibly using a programming language such as JQ) to define specific data manipulation routines.

[0105] In operation 710, the computing device integrates any optional custom transformations into the transformation process using the API. This step allows the system to process these user-defined transformations in parallel with the standard transformations, thereby enhancing the flexibility and capabilities of the data transformation module.

[0106] At operation 712, the computing device performs a conversion process. This step involves applying selected or customized conversion logic to the raw data, thereby converting it into the format required for its intended use.

[0107] At operation 714, the computing device verifies the converted data to ensure that it meets predefined standards. This verification checks the data for completeness, accuracy, and consistency with the conversion goals.

[0108] At operation 716, the computing device identifies errors or inconsistencies in the transformed data and takes necessary corrective actions. These may include reapplying the transformation with adjusted parameters or flagging the issue for human review and intervention.

[0109] At operation 718, the computing device formats the validated, transformed data into a final structure, making it ready for loading into a target system or for direct use. This step ensures that the data is not only accurate and relevant, but also presented in a manner that is accessible and understandable to the end user.

[0110] In operation 720, after successful conversion and formatting, the computing device outputs the data to be loaded into the designated destination system (e.g., a database or data warehouse). This marks the completion of the data conversion process and the transition to the final stage of the ETL workflow. Thus, method 700 implements a data conversion process within a codeless ETL system. The data involved in the process undergoes the necessary transformations while maintaining the integrity and specific requirements of the end-use application.

[0111] Figure 8800 is a block diagram of example components of a device 800. One or more computer systems 800 can be used, for example, to implement any of the embodiments described herein, and any combinations and subcombinations thereof. The computer system 800 can include one or more processors (also known as central processing units or CPUs), such as a processor 804. The processor 804 can be connected to a communication infrastructure or bus 806.

[0112] The computer system 800 may also include user input / output devices 803 , such as a monitor, keyboard, pointing device, etc., which may communicate with the communication infrastructure 806 through the user input / output interface 802 .

[0113] One or more processors 804 may be a graphics processing unit (GPU). In embodiments, a GPU may be a processor, i.e., a specialized electronic circuit designed to handle mathematically intensive applications. A GPU may have a parallel architecture that allows for efficient parallel processing of large blocks of data (e.g., mathematically intensive data commonly found in computer graphics applications), images, videos, and the like.

[0114] The computer system 800 may also include a main memory or primary storage 808, such as random access memory (RAM). The main memory 808 may include one or more levels of cache. The main memory 808 may store control logic (i.e., computer software) and / or data.

[0115] The computer system 800 may also include one or more secondary storage devices or memories 810. The secondary storage 810 may include, for example, a hard drive 812 and / or a removable storage device or drive 814.

[0116] The removable storage drive 814 can interact with a removable storage unit 818. The removable storage unit 818 can include a computer-usable or computer-readable storage device that stores computer software (control logic) and / or data. The removable storage unit 818 can be a program cartridge and cartridge interface (e.g., a program cartridge and cartridge interface in a video game device), a removable memory chip (e.g., an EPROM or PROM) and an associated socket, a memory stick and a USB port, a memory card and an associated memory card slot, and / or any other removable storage unit and associated interface. The removable storage drive 814 can read from and / or write to the removable storage unit 818.

[0117] Secondary memory 810 may include other devices, apparatuses, components, tools, or other means that allow computer system 800 to access computer programs and / or other instructions and / or data. Such devices, apparatuses, components, tools, or other means may include, for example, a removable storage unit 822 and an interface 820. Examples of removable storage unit 822 and interface 820 may include a program cartridge and a cartridge interface (e.g., a program cartridge and cartridge interface in a video game device), a removable memory chip (e.g., an EPROM or PROM) and an associated socket, a memory stick and a USB port, a memory card and an associated memory card slot, and / or any other removable storage unit and an associated interface.

[0118] The computer system 800 may also include a communication or network interface 824. The communication interface 824 may enable the computer system 800 to communicate and interact with any combination of external devices, external networks, external entities, and the like (individually and collectively designated by the reference numeral 828). For example, the communication interface 824 may allow the computer system 800 to communicate with an external or remote device 828 via a communication path 826, which may be wired and / or wireless (or a combination thereof) and may include any combination of a local area network, a wide area network, the Internet, and the like. Control logic and / or data may be transferred to / from the computer system 800 via the communication path 826.

[0119] The computer system 800 may also be any personal digital assistant (PDA), desktop workstation, laptop or notebook computer, netbook, tablet computer, smart phone, smart watch or other wearable device, appliance, local Internet of Things and / or embedded system, to name a few non-limiting examples, or any combination thereof.

[0120] The computer system 800 can be a client or server accessing or hosting any application and / or data through any delivery paradigm, including but not limited to remote or distributed cloud computing solutions; local or on-premises software ("local" cloud-based solutions); "as a service" models (e.g., Content as a Service (CaaS), Digital Content as a Service (DCaaS), Software as a Service (SaaS), Managed Software as a Service (MSaaS), Platform as a Service (PaaS), Desktop as a Service (DaaS), Framework as a Service (FaaS), Backend as a Service (BaaS), Mobile Backend as a Service (MBaaS), Infrastructure as a Service (IaaS), etc.); and / or a hybrid model including any combination of the foregoing examples or other services or delivery paradigms.

[0121] Any applicable data structures, file formats, and schemes in the computer system 800 may be derived from standards including, but not limited to, JavaScript Object Notation (JSON), Extensible Markup Language (XML), another markup language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), message packages, XML User Interface Language (XUL), or any other functionally similar representation (alone or in combination). Alternatively, proprietary data structures, formats, or schemes may be used alone or in combination with known or open standards.

[0122] In some embodiments, a tangible, non-transitory device or article of manufacture comprising a tangible, non-transitory computer-usable or readable medium storing control logic (software) may also be referred to herein as a computer program product or program storage device. This may include, but is not limited to, the computer system 800, primary memory 808, secondary memory 810, and removable storage units 818 and 822, as well as tangible articles of manufacture implementing any combination of the foregoing. Such control logic, when executed by one or more data processing devices (e.g., the computer system 800), may cause such data processing devices to operate as described herein.

[0123] It should be understood that the detailed description and the non-inventive summary and abstract sections are intended to be used to interpret the claims. The summary and abstract sections may set forth one or more, but not all, exemplary embodiments of the present invention contemplated by the inventors and are not intended to limit the present invention and the appended claims in any way.

[0124] The present invention is described above by means of functional building blocks that implement the specified functions and their relationships. For ease of description, the boundaries of these functional building blocks are arbitrarily defined herein. Alternative boundaries can be defined as long as the specified functions and their relationships are appropriately performed.

[0125] The description of the above specific embodiments will fully reveal the overall nature of the present invention. Without departing from the overall concept of the present invention, others can easily modify and / or rectify various applications (such as specific embodiments) by applying common sense of those skilled in the art without excessive experimentation. Therefore, based on the teachings and guidance proposed herein, such rectifications and modifications are intended to fall within the meaning and scope of the equivalent schemes of the disclosed embodiments. It should be understood that the wording or terminology herein is for descriptive and non-restrictive purposes, so that the terms or wording of this specification should be interpreted by the technician according to teaching and guidance.

[0126] The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A data processing system for automating an extraction, transformation, and loading (ETL) process, comprising: A server is coupled to the processor and configured to execute the following instructions: extracting data from a plurality of sources selected from one or more of cloud storage, an external API, and a direct file upload, wherein the extraction is performed by a stream mode processing unit configured to divide the data into two or more manageable chunks; verifying the structural integrity and content accuracy of the extracted data using a data integrity algorithm, said verification being defined by said data integrity algorithm based on at least one of: format consistency checking, anomaly detection, and data corruption identification; The extracted data is converted via a conversion processing unit, the conversion including application of predefined and custom conversion templates, the conversion processing unit being further configured to: Integration of user-defined transformation logic via API, which allows customization of data transformations; The converted data is formatted for loading into the destination system. 2 . The system according to claim 1 , wherein the stream mode processing unit dynamically adjusts the size of the data block based on the size of the data file and the system capacity. 3 . The system of claim 1 , wherein the data integrity algorithm includes an error correction mechanism configured to handle inconsistencies in the data identified during the extraction process.

4. The system of claim 1 , wherein instructions configure the processor to perform a codeless implementation of the automation, the codeless implementation comprising instructions wherein: The conversion processing unit utilizes a built-in programming language to define custom conversion logic, which enables users to define custom extensions in a low-code environment, thereby allowing the specification of custom operation rules, and The stream mode processing unit allows users to manage data block size and processing parameters without programming expertise.

5. The system of claim 1 , further comprising a no-code user interface configured to allow a user to select and configure transformations from a set of pre-built templates without programming expertise.

6. The system of claim 1, wherein the API for integrating user-defined transformation logic is compatible with a range of external programming environments. 7 . The system of claim 1 , wherein the server comprises a data loading module configured to load the converted data into two or more different destination systems, the destination systems being selected from a database and a data warehouse.

8. The system of claim 1, wherein the server is further configured to execute instructions for monitoring data flow throughout the ETL process, including tracking progress and performing real-time error correction.

9. The system according to claim 1 further comprises a recording module, wherein the recording module is configured to record one or more data change logs for storing changes to the data, including a timestamp, a nature of the change, and a user identification associated with the manual change, wherein the recording module enables full auditability and / or traceability of each change.

10. The system of claim 9, wherein the server comprises a rights management module configured to regulate access to the data change log and ensure traceability control.

11. A computer-implemented method comprising: identifying a data source for extraction, the data source selected from one or more of cloud storage, an external API, and a direct file upload; performing a data extraction process from the identified data source using a server coupled to the processor; performing a data integrity check on the extracted data using a data integrity algorithm to ensure structural accuracy and content consistency; configuring data transformations based on the validated data, including selecting from pre-built transformation templates and defining custom transformations; Integrate custom transformation logic into the data transformation process via an API; Apply the configured data transformations to the extracted data; validating the transformed data to ensure compliance with predefined standards; The validated and transformed data is formatted for loading into the target system.

12. The method of claim 11, wherein the data extraction process includes stream mode processing, and wherein the stream mode processing includes dividing the data into manageable chunks that are dynamically adjusted based on file size and / or system capacity.

13. The method of claim 11, wherein the data integrity check includes one or more error correction processes for resolving discrepancies identified in the extraction.

14. The method of claim 11, wherein configuring data transformations comprises implementing a built-in programming language for customizing transformation logic.

15. The method of claim 11, further comprising receiving input via a user interface regarding selecting and / or configuring one or more desired transformations from pre-built templates.

16. The method of claim 11, wherein integrating the custom transformation logic via an API comprises integrating the custom transformation logic via an API from one or more external programming environments.

17. The method of claim 11, wherein the method includes loading the transformed data into two or more different destination systems selected from a database and a data warehouse.

18. The method of claim 11, wherein the method includes monitoring data flow through the entire ETL process, tracking progress, and performing real-time error correction.

19. A non-transitory, tangible computer-readable device having instructions stored thereon, which, when executed by a computing device, cause the computing device to perform the following operations, the operations comprising: identifying a data source for extraction, the data source selected from one or more of cloud storage, an external API, and a direct file upload; Computer-based data extraction from identified data sources; Perform data integrity checks on the extracted data using data integrity algorithms to ensure structural accuracy and content consistency; Configure data transformations based on validated data, including selecting from pre-built transformation templates and defining custom transformations; Integrate custom transformation logic into the data conversion process via API; applying the configured data transformation to the extracted data; validating the transformed data to ensure compliance with predefined standards; The validated and transformed data is formatted for loading into the target system.

20. A computer-readable device according to claim 19, wherein performing the data extraction process includes performing streaming mode processing, wherein configuring data transformation includes utilizing a built-in programming language for defining custom transformation logic, wherein the custom transformation logic enables users to define custom extensions in a low-code environment, thereby allowing the specification of custom operation rules, and wherein performing the streaming mode processing includes dividing the data into manageable blocks that are dynamically adjusted based on file size and / or system capacity.

Images