Method, Apparatus, and System for Providing Data Refinement and Processing Solution with Improved Processing Speed for Large-Scale Data

Abstract

A method, device, and system for providing a data cleansing and processing solution that improves the processing speed of large-scale data are disclosed. In a method, apparatus, and system for providing a data refinement and processing solution with improved processing speed of large-capacity data according to an embodiment of the present invention, the apparatus comprises: a memory; a communication unit; The system includes one or more processors connected to the memory and communication unit, wherein the processor receives target data including large-capacity structured data and unstructured data from a user terminal or an external system, loads the target data into an internal data space within a cloud infrastructure without passing it to the application layer, and is configured as an in-memory execution environment based on a single address space where a storage area for data storage and a computation area for data computation are not logically separated. Based on the data loaded into the internal data space, it generates an index structure by analyzing the data structure in real time without undergoing serialization and deserialization processes, and databases the data without calling an external database. Furthermore, regarding the databased data, it directly executes data processing logic within the internal data space, including cleaning, filtering, aggregation, arithmetic operations, and statistical operations, without performing network-based data transmission and reception with an external application server or an external database server. The entire process from receiving the target data to database creation and data processing logic can be performed within a single execution environment.

Description

Technology Field

[0001] The present invention relates to a method, apparatus, and system for providing a data cleansing and processing solution that improves the processing speed of large-scale data in the fields of high-performance cloud computing and distributed database management systems (DBMS). More specifically, it relates to a method, apparatus, and system for providing a data cleansing and processing solution that improves the processing speed of large-scale data for high-speed processing of large-scale data sets including structured and unstructured data in a cloud computing environment and a distributed database management system (DBMS).

[0002] In particular, the method, device, and system for providing a data cleansing and processing solution that improves the processing speed of large-capacity data according to the present invention relates to a technology for improving problems such as data transmission delay, network bandwidth bottlenecks, and computational overhead in serialization and deserialization processes caused by a separated processing structure between the application layer and the data storage layer.

[0003] In addition, the method, device, and system for providing a data cleansing and processing solution that improves the processing speed of large-scale data according to the present invention relate to an integrated data processing architecture for efficiently performing a series of data processing processes, such as data collection, storage, cleansing, processing, and analysis, on a cloud-based virtualization infrastructure, and belong to a technical field that improves processing speed and system efficiency in a large-scale data processing environment by providing a structure that can perform computations while minimizing data movement.

[0004] Furthermore, the method, device, and system for providing a data refinement and processing solution with improved processing speed for large-scale data according to the present invention are applicable to industrial fields such as commerce, finance, and logistics that require real-time or near-real-time processing of large-scale transaction data, and in particular, relate to high-performance data processing technology required in systems that perform large-scale data settlement, aggregation, and analysis tasks. Background Technology

[0006] With the recent proliferation of e-commerce, digital platforms, and SaaS-based services, the volume of data that companies must process is increasing exponentially, and consequently, the importance of technologies for processing large volumes of data quickly and accurately is growing. In particular, in online shopping malls, marketplaces, and subscription services, various transaction data such as orders, payments, deliveries, refunds, and settlements are generated in real time, and this data must be aggregated and analyzed at regular intervals.

[0007] In such an environment, hundreds of thousands to millions of data records must be processed within a short period, and the system's processing performance directly impacts service quality, particularly when data processing requests are concentrated during specific timeframes, such as monthly closing or settlement periods.

[0008] Generally, existing data processing systems are configured with a hierarchical structure separating application servers and database servers. They operate by processing data on the application server and storing the results in the database, or conversely, querying data from the database to process it within the application. While this structure offers advantages in terms of system design flexibility and maintainability, it presents structural limitations in high-volume data processing environments.

[0009] First, to process large volumes of data, the process of transferring data from the database to the application server and saving it back to the database after processing is performed repeatedly. This leads to increased network bandwidth usage and accumulated communication delays. Particularly in cloud environments, even if such data transmission takes place over an internal network, the delays and costs associated with the data movement itself are significant and cannot be ignored.

[0010] Furthermore, serialization and deserialization are performed during the process of data moving between different layers; this consumes additional CPU and memory resources, thereby degrading overall processing performance. This overhead increases even further, particularly when using structured data formats such as JSON and XML.

[0011] Furthermore, the separation of roles between the application and database layers leads to the dispersion of data processing logic, which in turn complicates the processing flow and makes inter-system synchronization and transaction management difficult. In such an architecture, it is difficult to track errors or bottlenecks occurring during data processing, and this can also affect the stability and reliability of the entire system.

[0012] Furthermore, as the scale of data increases, processing time does not simply rise linearly but tends to increase inefficiently due to factors such as data movement costs, processing overhead, and bottlenecks. In particular, when processing large volumes of data in batch mode, the total processing time can exceed tens of minutes, which acts as a significant constraint in service environments requiring real-time or near-real-time processing.

[0013] Meanwhile, various approaches such as in-memory databases, distributed processing frameworks, and parallel processing technologies have been proposed to address these issues. For example, in-memory based processing methods improve data processing speed by minimizing disk access, while distributed processing methods provide the effect of improving processing performance by distributing tasks across multiple nodes.

[0014] However, these technologies also fail to completely eliminate the structure in which data moves between different processing entities, and overhead still exists in the data transmission and processing process. Furthermore, in distributed processing environments, additional complexities such as inter-node communication costs, maintaining data consistency, and fault handling may arise, and there is also the issue of increased system operation and management costs.

[0015] In particular, in environments such as the commerce industry where data must be collected from various external platforms (e.g., marketplaces, payment systems, etc.) and integrated for processing, additional data cleaning and transformation processes are required due to the diversity and unstructured nature of data formats. This leads to problems where the entire processing pipeline becomes more complex and processing time increases.

[0016] Consequently, existing data processing methods simultaneously entail problems such as data movement, processing overhead, system complexity, and scalability limitations in large-scale data environments. To fundamentally resolve these issues, a new type of data processing structure is required that minimizes data movement and enables computations to be performed directly at the location where the data resides.

[0017] Therefore, a technical approach is required to improve overall processing efficiency by more closely combining data storage and computation and performing the data processing process within a single execution environment, and the present invention is proposed as a solution to this. The problem to be solved

[0019] The present invention has been devised to solve the problems of the aforementioned prior art, and aims to provide a method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data, thereby minimizing delays caused by repetitive data movement between a server and a database during the large-scale data processing process and improving data processing efficiency.

[0020] In addition, the present invention aims to provide a method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-capacity data, comprising a structure that enables data processing to be performed without or with minimal movement of data by reducing the physical or logical separation between the location where data is stored and the location where operations are performed, and by allowing data processing logic to be performed in the same area where data is stored or in an adjacent execution environment.

[0021] In addition, the present invention aims to provide a method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data, enabling various data processing tasks including refinement, processing, filtering, and arithmetic operations of large-scale data to be performed efficiently within a single processing flow, reducing input / output bottlenecks occurring during the data processing process, and maximizing the utilization of system resources.

[0022] In addition, the present invention aims to provide a method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data, comprising a scalable data processing structure that allows system resources to be flexibly expanded in accordance with the increase in data size, thereby maintaining stable processing performance despite the increase in data scale.

[0023] Furthermore, the present invention aims to provide a method, apparatus, and system for providing a data refinement and processing solution with improved processing speed for large-scale data, which can simultaneously ensure high-speed processing and stability while satisfying various types of data processing requirements in a cloud-based environment. means of solving the problem

[0025] In a method, apparatus, and system for providing a data refinement and processing solution with improved processing speed of large-capacity data according to an embodiment of the present invention, the apparatus comprises: a memory; a communication unit; The system includes one or more processors connected to the memory and communication unit, wherein the processor receives target data including large-capacity structured data and unstructured data from a user terminal or an external system, loads the target data into an internal data space within a cloud infrastructure without passing it to the application layer, and is configured as an in-memory execution environment based on a single address space where a storage area for data storage and a computation area for data computation are not logically separated. Based on the data loaded into the internal data space, it generates an index structure by analyzing the data structure in real time without undergoing serialization and deserialization processes, and databases the data without calling an external database. Furthermore, regarding the databased data, it directly executes data processing logic within the internal data space, including cleaning, filtering, aggregation, arithmetic operations, and statistical operations, without performing network-based data transmission and reception with an external application server or an external database server. The entire process from receiving the target data to database creation and data processing logic can be performed within a single execution environment.

[0026] According to one embodiment of the present invention, the internal data space comprises a plurality of physical nodes, and each node includes a distributed shared memory structure that includes independent local memory while logically sharing a single address space. The processor analyzes the structure, size, and memory access pattern of data loaded in the internal data space and divides the data processing task into a plurality of task units created in units of data blocks or memory pages. The processor schedules the plurality of task units to minimize local memory access by considering the cache structure and NUMA (Non-Uniform Memory Access) characteristics of each CPU core. The processor executes the plurality of task units in parallel on an in-memory based parallel processing pipeline. During the parallel execution of the plurality of task units, data exchange between nodes is performed using a memory mapping-based shared memory access method without using network protocol-based communication, thereby restricting data movement to occur only within the memory layer.

[0027] According to one embodiment of the present invention, the processor connects a plurality of transaction data based on the same identifier based on the result of data processing by data processing logic within the internal data space, analyzes a repurchase pattern including data sorting in chronological order, identifier-based grouping, and determination of whether to make repeated purchases based on the connection structure of the plurality of transaction data, determines whether the data value surges or plummets using an average value, standard deviation, quantile, or threshold value based on the time series distribution of the data according to the repurchase pattern, and generates insight data including the result of determining whether to surge or plummet. The process of generating the insight data is performed within the internal data space without calling an external system, and the insight data is generated without additional data movement and provided to a user terminal, thereby providing real-time or near-real-time analysis results simultaneously with data processing.

[0028] According to one embodiment of the present invention, the memory includes a first model that has been trained to predict at least one purchase scenario for input target data and to calculate the probability of occurrence of the predicted purchase scenario; the processor trains the first model to predict at least one purchase scenario and the probability of occurrence of the purchase scenario based on the target data by utilizing the time-series order of identifiers included in the target data and combinations of identifiers based on the accumulated target data; takes the target data as input and predicts a judgment result regarding whether the target data will surge or plummet based on the output result of the first model; provides the predicted judgment result to a user terminal; compares the insight data with the prediction result of the first model; if a discrepancy in whether the target data will surge or plummet occurs or the difference in the calculated degree of fluctuation exceeds a threshold, the prediction result and the target data are stored as noise data; analyzes the number of noise occurrences based on the identifiers included in the accumulated noise data; identifies at least one common identifier among the identifiers included in the accumulated noise data whose number of noise occurrences is greater than or equal to a threshold; and if the common identifier is identified, the number of noise occurrences is inversely proportional to the number of noise occurrences for the common identifier A first weight is calculated, the first weight is stored corresponding to a common identifier, and when new target data including the common identifier is received, the first weight is applied to correct and adjust the probability of a purchase scenario occurring for the new target data of the first model in a decreasing direction.

[0029] The processor according to an embodiment of the present invention classifies a plurality of target data based on a plurality of occurrence addresses where data occurred, calculates a purchase scenario and a purchase probability for each occurrence address based on a plurality of first models configured independently for each occurrence address, calculates a purchase probability for each of a plurality of identifiers based on an occurrence address-purchase item based on an actual purchase history for each occurrence address, calculates an average purchase probability for each of the plurality of identifiers based on at least one common purchase item which is a purchase item sold identically for the plurality of occurrence addresses, calculates a ranking of the purchase probability of the plurality of identifiers for each common purchase item, and, based on a top identifier among the plurality of identifiers that is above a threshold rank within the ranking, compares the difference between the purchase probability for each occurrence address corresponding to the top identifier and the average purchase probability, identifies a first occurrence address corresponding to a purchase probability lower than the average purchase probability by a threshold or higher, and a second occurrence address corresponding to a purchase probability higher than the average purchase probability, selects at least one first purchase scenario including the top identifier for the first occurrence address, selects at least one second purchase scenario including the top identifier for the second occurrence address, and the first purchase Using a scenario and the second purchase scenario as input, a third model is generated or trained to correct the output result of a second model that has been previously trained for the first occurrence address among the plurality of first models, and the correction value calculated by the third model is applied to the output of the second model to generate a corrected purchase scenario occurrence probability with the bias per occurrence address removed, and based on the corrected purchase scenario occurrence probability, a result of determining whether the common purchase item of the first occurrence address has surged or plummeted is recalculated, and the third model comprises the first purchase scenario, the second purchase scenario,The artificial intelligence model is trained to calculate a correction value by reflecting the deviation by occurrence address on the purchase scenario occurrence probability generated as the output of the second model, using as training data the purchase probability by occurrence address for the above common purchase item, the average purchase probability and deviation value by occurrence address, and correct data regarding whether there is a surge or a surge based on actual purchase history, and the plurality of occurrence addresses may include user terminals or external systems.

[0030] A device according to one embodiment of the present invention may be combined with hardware and controlled by a computer program stored in a medium to execute any one of the operation methods by the processor of the system described above. Effects of the invention

[0032] According to the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-capacity data according to the present invention, delays caused by data movement can be reduced by minimizing the separation between the area where data is stored and the area where operations are performed.

[0033] Accordingly, the upload, refinement, and processing of large-scale data are shortened compared to existing methods, and stable processing performance can be maintained, especially even when processing data on the scale of hundreds of thousands or millions of records.

[0034] In addition, according to the method, apparatus, and system for providing a data cleansing and processing solution that improves the processing speed of large-capacity data according to the present invention, as the data processing process is performed within a single integrated execution environment, the overhead generated during the data serialization and deserialization processes is reduced, and the efficiency of the entire data processing pipeline can be improved.

[0035] In addition, according to the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-capacity data according to the present invention, by applying in-memory based processing and parallel processing techniques, data access speed can be improved and I / O bottlenecks between systems can be alleviated, thereby maintaining stable performance even in an environment where a large volume of transactions occur simultaneously.

[0036] In addition, according to the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-capacity data according to the present invention, resources required for data processing are configured to dynamically adjust according to the size and throughput of the data, thereby enabling efficient use of system resources and reducing unnecessary infrastructure costs.

[0037] In particular, according to the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data according to the present invention, in an environment where large-scale data processing is concentrated at the time of settlement and closing, such as in the commerce industry, processing delays and system failures can be prevented and stable service provision can be enabled, thereby improving user experience and service reliability.

[0038] Consequently, the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data according to the present invention provide the effect of simultaneously securing processing speed, stability, and scalability in a large-scale data processing environment. Brief explanation of the drawing

[0040] FIG. 1 is a block diagram schematically illustrating the basic configuration of a device according to one embodiment. FIG. 2 is a schematic diagram illustrating a system according to one embodiment. FIGS. 3 to 5 are basic operation flowcharts of a method according to one embodiment. FIG. 6 is an additional operation flowchart of a method according to another embodiment. Specific details for implementing the invention

[0041] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, various modifications may be made to the embodiments, and thus the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, and substitutions to the embodiments are included within the scope of the rights.

[0042] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed forms, and the scope of this specification includes modifications, equivalents, or substitutions that fall within the technical concept.

[0043] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0044] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0045] The terms used in the embodiments are for illustrative purposes only and should not be interpreted as intended to be limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0046] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the embodiments pertain. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0047] In addition, when describing with reference to the attached drawings, identical components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the embodiments, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the embodiments, such detailed description is omitted.

[0048] The embodiments can be implemented in various forms of products such as personal computers, laptop computers, tablet computers, smartphones, televisions, smart home appliances, intelligent automobiles, kiosks, and wearable devices.

[0049] Artificial Intelligence (AI) systems are computer systems that implement human-level intelligence; unlike existing rule-based smart systems, they are systems in which machines learn and make decisions autonomously. As AI systems improve in recognition accuracy and gain a more accurate understanding of user preferences with continued use, existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

[0050] Artificial intelligence technology consists of machine learning and component technologies utilizing machine learning. Machine learning is an algorithmic technology that autonomously classifies and learns the characteristics of input data, while component technologies are technologies that mimic the cognitive and judgmental functions of the human brain by utilizing machine learning algorithms such as deep learning, and are comprised of technological fields such as linguistic understanding, visual understanding, reasoning / prediction, knowledge representation, and motion control.

[0051] The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding refers to technologies that recognize, apply, and process human language and text, including natural language processing, machine translation, dialogue systems, question answering, and speech recognition / synthesis. Visual understanding refers to technologies that perceive and process objects like human vision, including object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, and image enhancement. Inference and prediction refers to technologies that logically reason and predict by judging information, including knowledge / probability-based inference, optimization prediction, preference-based planning, and recommendation. Knowledge representation refers to technologies that automatically process human experiential information into knowledge data, including knowledge construction (data generation / classification) and knowledge management (data utilization). Motion control refers to technologies that control the autonomous driving of vehicles and the movement of robots, including motion control (navigation, collision, driving) and manipulation control (behavior control).

[0052] Generally, to apply machine learning algorithms to real-world situations, training is performed using a trial-and-error method due to the inherent characteristics of the fundamental methodologies. In particular, deep learning requires hundreds of thousands of iterations. Since it is impossible to execute this in a real physical external environment, training is instead performed through simulations that virtually recreate the actual physical environment on a computer.

[0053] In the present invention, Artificial Intelligence (AI) refers to a technology that imitates human learning ability, reasoning ability, and perceptual ability, and implements them on a computer, and may include concepts such as machine learning and symbolic logic. Machine Learning (ML) is an algorithmic technology that classifies or learns the characteristics of input data on its own. AI technology can analyze input data as a machine learning algorithm, learn from the results of the analysis, and make judgments or predictions based on the results of the learning. Furthermore, technologies that mimic the functions of the human brain, such as cognition and judgment, by utilizing machine learning algorithms can also be understood as falling within the category of AI. For example, technological fields such as linguistic understanding, visual understanding, reasoning / prediction, knowledge representation, and motion control may be included.

[0054] Machine learning can refer to the process of training neural network models using experience in processing data. Through machine learning, computer software can improve its own data processing capabilities. A neural network model is constructed by modeling the correlations between data, and these correlations can be expressed by multiple parameters. A neural network model extracts and analyzes features from given data to derive correlations between them; machine learning can be defined as the process of optimizing the model's parameters by repeating this process. For example, a neural network model can learn the mapping (correlation) between inputs and outputs for data given as input-output pairs. Alternatively, even when only input data is provided, a neural network model can derive regularities between the given data and learn those relationships.

[0055] An artificial intelligence learning model or neural network model can be designed to implement the structure of the human brain on a computer and may include multiple network nodes that have weights and simulate neurons of a human neural network. The multiple network nodes may have interconnected relationships by simulating the synaptic activity of neurons, where neurons exchange signals through synapses. In an artificial intelligence learning model, multiple network nodes may be located in layers of different depths and exchange data according to convolutional connections. The artificial intelligence learning model may be, for example, an Artificial Neural Network (ANN) or a Convolutional Neural Network (CNN). As an embodiment, the artificial intelligence learning model may be machine learned according to methods such as supervised learning, unsupervised learning, and reinforcement learning. Machine learning algorithms for performing machine learning may include Decision Tree, Bayesian Network, Support Vector Machine, Artificial Neural Network, Ada-boost, Perceptron, Genetic Programming, and Clustering.

[0056] Among these, CNNs are a type of multilayer perceptron designed to use minimal preprocessing. CNNs consist of one or more convolutional layers and standard artificial neural network layers stacked on top, additionally utilizing weights and pooling layers. Thanks to this structure, CNNs can fully utilize two-dimensional input data. Compared to other deep learning architectures, CNNs demonstrate good performance in both image and audio fields. CNNs can also be trained using standard backpropagation. CNNs have the advantage of being easier to train than other feedforward artificial neural network techniques and using a small number of parameters.

[0057] Convolutional networks are neural networks comprising sets of nodes with bounded parameters. Many computer vision tasks have been significantly improved, driven by the increased size of available training data and the availability of computational power, combined with algorithmic advancements such as discriminative linear units and dropout training. In the case of massive datasets, such as those available for many tasks today, outfitting is not critical, and increasing the network size improves test accuracy. Optimal utilization of computing resources becomes a limiting factor. To address this, distributed, scalable implementations of deep neural networks can be employed.

[0058] The present invention relates to a method, apparatus, and system for providing a data cleansing and processing solution that improves the processing speed of large-scale data in the fields of high-performance cloud computing and distributed database management systems (DBMS). More specifically, it relates to a method, apparatus, and system for providing a data cleansing and processing solution that improves the processing speed of large-scale data for high-speed processing of large-scale data sets including structured and unstructured data in a cloud computing environment and a distributed database management system (DBMS).

[0059] In particular, the method, device, and system for providing a data cleansing and processing solution that improves the processing speed of large-capacity data according to the present invention relates to a technology for improving problems such as data transmission delay, network bandwidth bottlenecks, and computational overhead in serialization and deserialization processes caused by a separated processing structure between the application layer and the data storage layer.

[0060] In addition, the method, device, and system for providing a data cleansing and processing solution that improves the processing speed of large-scale data according to the present invention relate to an integrated data processing architecture for efficiently performing a series of data processing processes, such as data collection, storage, cleansing, processing, and analysis, on a cloud-based virtualization infrastructure, and belong to a technical field that improves processing speed and system efficiency in a large-scale data processing environment by providing a structure that can perform computations while minimizing data movement.

[0061] Furthermore, the method, device, and system for providing a data refinement and processing solution with improved processing speed for large-scale data according to the present invention are applicable to industrial fields such as commerce, finance, and logistics that require real-time or near-real-time processing of large-scale transaction data, and in particular, relate to high-performance data processing technology required in systems that perform large-scale data settlement, aggregation, and analysis tasks.

[0062] With the recent proliferation of e-commerce, digital platforms, and SaaS-based services, the volume of data that companies must process is increasing exponentially, and consequently, the importance of technologies for processing large volumes of data quickly and accurately is growing. In particular, in online shopping malls, marketplaces, and subscription services, various transaction data such as orders, payments, deliveries, refunds, and settlements are generated in real time, and this data must be aggregated and analyzed at regular intervals.

[0063] In such an environment, hundreds of thousands to millions of data records must be processed within a short period, and the system's processing performance directly impacts service quality, particularly when data processing requests are concentrated during specific timeframes, such as monthly closing or settlement periods.

[0064] Generally, existing data processing systems are configured with a hierarchical structure separating application servers and database servers. They operate by processing data on the application server and storing the results in the database, or conversely, querying data from the database to process it within the application. While this structure offers advantages in terms of system design flexibility and maintainability, it presents structural limitations in high-volume data processing environments.

[0065] First, to process large volumes of data, the process of transferring data from the database to the application server and saving it back to the database after processing is performed repeatedly. This leads to increased network bandwidth usage and accumulated communication delays. Particularly in cloud environments, even if such data transmission takes place over an internal network, the delays and costs associated with the data movement itself are significant and cannot be ignored.

[0066] Furthermore, serialization and deserialization are performed during the process of data moving between different layers; this consumes additional CPU and memory resources, thereby degrading overall processing performance. This overhead increases even further, particularly when using structured data formats such as JSON and XML.

[0067] Furthermore, the separation of roles between the application and database layers leads to the dispersion of data processing logic, which in turn complicates the processing flow and makes inter-system synchronization and transaction management difficult. In such an architecture, it is difficult to track errors or bottlenecks occurring during data processing, and this can also affect the stability and reliability of the entire system.

[0068] Furthermore, as the scale of data increases, processing time does not simply rise linearly but tends to increase inefficiently due to factors such as data movement costs, processing overhead, and bottlenecks. In particular, when processing large volumes of data in batch mode, the total processing time can exceed tens of minutes, which acts as a significant constraint in service environments requiring real-time or near-real-time processing.

[0069] Meanwhile, various approaches such as in-memory databases, distributed processing frameworks, and parallel processing technologies have been proposed to address these issues. For example, in-memory based processing methods improve data processing speed by minimizing disk access, while distributed processing methods provide the effect of improving processing performance by distributing tasks across multiple nodes.

[0070] However, these technologies also fail to completely eliminate the structure in which data moves between different processing entities, and overhead still exists in the data transmission and processing process. Furthermore, in distributed processing environments, additional complexities such as inter-node communication costs, maintaining data consistency, and fault handling may arise, and there is also the issue of increased system operation and management costs.

[0071] In particular, in environments such as the commerce industry where data must be collected from various external platforms (e.g., marketplaces, payment systems, etc.) and integrated for processing, additional data cleaning and transformation processes are required due to the diversity and unstructured nature of data formats. This leads to problems where the entire processing pipeline becomes more complex and processing time increases.

[0072] Consequently, existing data processing methods simultaneously entail problems such as data movement, processing overhead, system complexity, and scalability limitations in large-scale data environments. To fundamentally resolve these issues, a new type of data processing structure is required that minimizes data movement and enables computations to be performed directly at the location where the data resides.

[0073] Therefore, a technical approach is required to improve overall processing efficiency by more closely combining data storage and computation and performing the data processing process within a single execution environment, and the present invention is proposed as a solution to this.

[0074] The present invention has been devised to solve the problems of the aforementioned prior art, and aims to provide a method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data, thereby minimizing delays caused by repetitive data movement between a server and a database during the large-scale data processing process and improving data processing efficiency.

[0075] In addition, the present invention aims to provide a method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-capacity data, comprising a structure that enables data processing to be performed without or with minimal movement of data by reducing the physical or logical separation between the location where data is stored and the location where operations are performed, and by allowing data processing logic to be performed in the same area where data is stored or in an adjacent execution environment.

[0076] In addition, the present invention aims to provide a method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data, enabling various data processing tasks including refinement, processing, filtering, and arithmetic operations of large-scale data to be performed efficiently within a single processing flow, reducing input / output bottlenecks occurring during the data processing process, and maximizing the utilization of system resources.

[0077] In addition, the present invention aims to provide a method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data, comprising a scalable data processing structure that allows system resources to be flexibly expanded in accordance with the increase in data size, thereby maintaining stable processing performance despite the increase in data scale.

[0078] Furthermore, the present invention aims to provide a method, apparatus, and system for providing a data refinement and processing solution with improved processing speed for large-scale data, which can simultaneously ensure high-speed processing and stability while satisfying various types of data processing requirements in a cloud-based environment.

[0079] In a method, apparatus, and system for providing a data refinement and processing solution with improved processing speed of large-capacity data according to an embodiment of the present invention, the apparatus comprises: a memory; a communication unit; The system includes one or more processors connected to the memory and communication unit, wherein the processor receives target data including large-capacity structured data and unstructured data from a user terminal or an external system, loads the target data into an internal data space within a cloud infrastructure without passing it to the application layer, and is configured as an in-memory execution environment based on a single address space where a storage area for data storage and a computation area for data computation are not logically separated. Based on the data loaded into the internal data space, it generates an index structure by analyzing the data structure in real time without undergoing serialization and deserialization processes, and databases the data without calling an external database. Furthermore, regarding the databased data, it directly executes data processing logic within the internal data space, including cleaning, filtering, aggregation, arithmetic operations, and statistical operations, without performing network-based data transmission and reception with an external application server or an external database server. The entire process from receiving the target data to database creation and data processing logic can be performed within a single execution environment.

[0080] According to one embodiment of the present invention, the internal data space comprises a plurality of physical nodes, and each node includes a distributed shared memory structure that includes independent local memory while logically sharing a single address space. The processor analyzes the structure, size, and memory access pattern of data loaded in the internal data space and divides the data processing task into a plurality of task units created in units of data blocks or memory pages. The processor schedules the plurality of task units to minimize local memory access by considering the cache structure and NUMA (Non-Uniform Memory Access) characteristics of each CPU core. The processor executes the plurality of task units in parallel on an in-memory based parallel processing pipeline. During the parallel execution of the plurality of task units, data exchange between nodes is performed using a memory mapping-based shared memory access method without using network protocol-based communication, thereby restricting data movement to occur only within the memory layer.

[0081] According to one embodiment of the present invention, the processor connects a plurality of transaction data based on the same identifier based on the result of data processing by data processing logic within the internal data space, analyzes a repurchase pattern including data sorting in chronological order, identifier-based grouping, and determination of whether to make repeated purchases based on the connection structure of the plurality of transaction data, determines whether the data value surges or plummets using an average value, standard deviation, quantile, or threshold value based on the time series distribution of the data according to the repurchase pattern, and generates insight data including the result of determining whether to surge or plummet. The process of generating the insight data is performed within the internal data space without calling an external system, and the insight data is generated without additional data movement and provided to a user terminal, thereby providing real-time or near-real-time analysis results simultaneously with data processing.

[0082] According to one embodiment of the present invention, the memory includes a first model that has been trained to predict at least one purchase scenario for input target data and to calculate the probability of occurrence of the predicted purchase scenario; the processor trains the first model to predict at least one purchase scenario and the probability of occurrence of the purchase scenario based on the target data by utilizing the time-series order of identifiers included in the target data and combinations of identifiers based on the accumulated target data; takes the target data as input and predicts a judgment result regarding whether the target data will surge or plummet based on the output result of the first model; provides the predicted judgment result to a user terminal; compares the insight data with the prediction result of the first model; if a discrepancy in whether the target data will surge or plummet occurs or the difference in the calculated degree of fluctuation exceeds a threshold, the prediction result and the target data are stored as noise data; analyzes the number of noise occurrences based on the identifiers included in the accumulated noise data; identifies at least one common identifier among the identifiers included in the accumulated noise data whose number of noise occurrences is greater than or equal to a threshold; and if the common identifier is identified, the number of noise occurrences is inversely proportional to the number of noise occurrences for the common identifier A first weight is calculated, the first weight is stored corresponding to a common identifier, and when new target data including the common identifier is received, the first weight is applied to correct and adjust the probability of a purchase scenario occurring for the new target data of the first model in a decreasing direction.

[0083] The processor according to an embodiment of the present invention classifies a plurality of target data based on a plurality of occurrence addresses where data occurred, calculates a purchase scenario and a purchase probability for each occurrence address based on a plurality of first models configured independently for each occurrence address, calculates a purchase probability for each of a plurality of identifiers based on an occurrence address-purchase item based on an actual purchase history for each occurrence address, calculates an average purchase probability for each of the plurality of identifiers based on at least one common purchase item which is a purchase item sold identically for the plurality of occurrence addresses, calculates a ranking of the purchase probability of the plurality of identifiers for each common purchase item, and, based on a top identifier among the plurality of identifiers that is above a threshold rank within the ranking, compares the difference between the purchase probability for each occurrence address corresponding to the top identifier and the average purchase probability, identifies a first occurrence address corresponding to a purchase probability lower than the average purchase probability by a threshold or higher, and a second occurrence address corresponding to a purchase probability higher than the average purchase probability, selects at least one first purchase scenario including the top identifier for the first occurrence address, selects at least one second purchase scenario including the top identifier for the second occurrence address, and the first purchase Using a scenario and the second purchase scenario as input, a third model is generated or trained to correct the output result of a second model that has been previously trained for the first occurrence address among the plurality of first models, and the correction value calculated by the third model is applied to the output of the second model to generate a corrected purchase scenario occurrence probability with the bias per occurrence address removed, and based on the corrected purchase scenario occurrence probability, a result of determining whether the common purchase item of the first occurrence address has surged or plummeted is recalculated, and the third model comprises the first purchase scenario, the second purchase scenario,The artificial intelligence model is trained to calculate a correction value by reflecting the deviation by occurrence address on the purchase scenario occurrence probability generated as the output of the second model, using as training data the purchase probability by occurrence address for the above common purchase item, the average purchase probability and deviation value by occurrence address, and correct data regarding whether there is a surge or a surge based on actual purchase history, and the plurality of occurrence addresses may include user terminals or external systems.

[0084] A device according to one embodiment of the present invention may be combined with hardware and controlled by a computer program stored in a medium to execute any one of the operation methods by the processor of the system described above.

[0085] According to the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-capacity data according to the present invention, delays caused by data movement can be reduced by minimizing the separation between the area where data is stored and the area where operations are performed.

[0086] Accordingly, the upload, refinement, and processing of large-scale data are shortened compared to existing methods, and stable processing performance can be maintained, especially even when processing data on the scale of hundreds of thousands or millions of records.

[0087] In addition, according to the method, apparatus, and system for providing a data cleansing and processing solution that improves the processing speed of large-capacity data according to the present invention, as the data processing process is performed within a single integrated execution environment, the overhead generated during the data serialization and deserialization processes is reduced, and the efficiency of the entire data processing pipeline can be improved.

[0088] In addition, according to the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-capacity data according to the present invention, by applying in-memory based processing and parallel processing techniques, data access speed can be improved and I / O bottlenecks between systems can be alleviated, thereby maintaining stable performance even in an environment where a large volume of transactions occur simultaneously.

[0089] In addition, according to the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-capacity data according to the present invention, resources required for data processing are configured to dynamically adjust according to the size and throughput of the data, thereby enabling efficient use of system resources and reducing unnecessary infrastructure costs.

[0090] In particular, according to the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data according to the present invention, in an environment where large-scale data processing is concentrated at the time of settlement and closing, such as in the commerce industry, processing delays and system failures can be prevented and stable service provision can be enabled, thereby improving user experience and service reliability.

[0091] Consequently, the method, apparatus, and system for providing a data refinement and processing solution that improves the processing speed of large-scale data according to the present invention provide the effect of simultaneously securing processing speed, stability, and scalability in a large-scale data processing environment.

[0092] FIG. 1 is a block diagram schematically illustrating the basic configuration of a device according to one embodiment, and FIG. 2 is a diagram schematically illustrating a system according to one embodiment.

[0093] Referring to FIG. 1, the device (100) of the present invention may be an electronic device (100) comprising a processor (110) and a memory (120). An electronic device (100) according to one embodiment may be a server or a terminal. According to one embodiment, the processor (110) may be composed of one or more processors (110) such that it is configured to perform operations or data processing regarding the control and / or communication of each component of the electronic device (100). The memory (120) may store information related to the method described above or store a program in which the method described above is implemented. The memory (120) may be a volatile memory (120) or a non-volatile memory (120).

[0094] According to one embodiment, the processor (110) can execute a program and control a device. The code of the program executed by the processor (110) can be stored in memory (120). Operations of the processor (110) can be performed by loading instructions stored in memory (120). The electronic device (100) can be connected to an external device (e.g., a personal computer or a network) through an input / output device (not shown in the drawing) and exchange data.

[0095] According to one embodiment, there are no limitations on the computation and data processing functions that the processor (110) can implement on the electronic device (100), but the process processing functions of the system of the present invention will be described below.

[0096] Additionally, the electronic device (100) according to one embodiment may further include a communication unit (130).

[0097] An electronic device (100) is connected to a communication unit (130) and, as shown in FIG. 2, can transmit and receive signals and / or data to and from another electronic device (e.g., user terminal, 20) and another server (e.g., external system, 10) through the communication unit (130). The communication unit (130) may be provided with a conductive line or a wireless communication chip, etc.

[0098] In FIG. 2, the external system (10) and the user terminal (20) are each shown as one, but this is merely an example and is not limited thereto, and it is obvious that multiple ones can be provided.

[0099] Meanwhile, the present invention relates to an integrated data processing framework comprising: a first system that generates an execution graph by metadataizing API specifications (user manuals for calling external systems) and unstructured data structures, and dynamically configures and executes functional units at runtime based on the execution graph; and a second system that predicts the scope of impact and corrects the execution engine in response to changes in the execution graph and updates to external systems.

[0100] Here, unstructured data structures are represented in the form of JSON, XML, log text, or key-value pairs, and can refer to data with variable field configuration and hierarchy without a fixed schema.

[0101] According to the present invention, by configuring data processing execution and response to changes in execution structure into separate layers, only the partial scope of impact can be dynamically corrected without reconfiguring the entire system even when an external system is changed; accordingly, execution stability and data processing accuracy can be simultaneously improved while maintaining real-time processing continuity without system interruption.

[0102] The first system may be configured as an execution-oriented structure responsible for data processing and function execution, and the second system may be configured as an independent management layer that performs change impact analysis and correction on the execution structure of the first system.

[0103] Here, an execution-oriented structure refers to a structure in which data storage, processing, and analysis functions are not separated but are performed integrally within a single memory-based execution environment; this enables improved processing speed by eliminating delays that occur during data movement and serialization.

[0104] The first system and the second system may be configured as logically separated functional modules, but may be physically implemented integrally by one or more processors, and the operation by the processor (110) will be described below.

[0105] Hereinafter, the operation of the processor (110) can be described as the operation of the electronic device (100). Additionally, each operation of the processor (110) can be implemented in the form of a software module and executed sequentially or in parallel by the processor.

[0106] FIGS. 3 to 5 are basic operation flowcharts of a method according to one embodiment.

[0107] Referring to FIG. 3, the processor (110) can receive target data (S310) from a user terminal (20) or an external system (10).

[0108] For example, the target data may include large amounts of structured and unstructured data.

[0109] The processor (110) can load the target data into the internal data space (S320) without passing it to the application layer.

[0110] For example, the internal data space may be a space configured as an in-memory execution environment based on a single address space within the cloud infrastructure, in which a storage area for data storage and a computation area for data computation are not logically separated. As an example, the internal data space may include a distributed shared memory (120) structure composed of multiple physical nodes, each node containing independent local memory while logically sharing a single address space.

[0111] Due to this structure, data moves only within the memory layer without external transmission over the network, so data processing delays and network bottlenecks are minimized, and real-time processing performance for large volumes of data can be improved.

[0112] The processor (110) can create an index structure by analyzing the data structure in real time (S330) based on the data loaded in the internal data space without undergoing serialization and deserialization processes, and can database (S340) without calling an external database.

[0113] This eliminates the ETL (Extract-Transform-Load) or serialization-based storage procedures that are essential in the existing database construction process, which can simplify the data processing pipeline and significantly reduce processing delays.

[0114] The processor (110) can directly execute data processing logic (S350) within the internal data space without performing network-based data transmission and reception with an external application server or an external database server for the databased data.

[0115] For example, data processing logic may include cleaning, filtering, aggregation, arithmetic operations, and statistical operations.

[0116] In the entire process from receiving the aforementioned target data to database creation and data processing logic, the processor (110) can perform the processes within a single execution environment.

[0117] Referring to FIGS. 3 and FIGS. 4, a processor (110) according to one embodiment of the present invention can analyze (S410) the structure, size, and memory access pattern of data loaded in an internal data space while performing step S330.

[0118] Accordingly, with reference to FIGS. 3 to 5, in performing step S350, the processor (110) can divide the data work by the data processing logic into a plurality of task units (S510) generated in units of data blocks or memory pages.

[0119] The processor (110) can schedule multiple work units to minimize local memory access by considering the cache structure and NUMA (Non-Uniform Memory Access) characteristics of each CPU core (e.g., the process of distributing which work to execute on which core and when) (S520). For example, scheduling to minimize local memory access can be configured so that the processor (110) schedules each work unit to preferentially reference a memory area corresponding to the CPU core.

[0120] The processor (110) can execute multiple work units in parallel (S530) on an in-memory based parallel processing pipeline.

[0121] At this time, while the processor (110) executes multiple work units in parallel, data exchange between nodes is performed using a memory (120) mapping-based shared memory (120) access method without using network protocol-based communication, thereby restricting data movement to occur only within the memory (120) layer.

[0122] As a result, communication costs between nodes are eliminated and CPU cache utilization increases, which can improve parallel processing efficiency and overall throughput.

[0123] Meanwhile, a processor (110) according to one embodiment of the present invention can connect multiple transaction data based on the same identifier based on the result of data processing by data processing logic within an internal data space.

[0124] The processor (110) can analyze repurchase patterns based on the connection structure of multiple transaction data.

[0125] For example, repurchase patterns may include sorting data in chronological order, grouping based on identifiers, and determining whether to make repeat purchases.

[0126] The processor (110) can determine whether the data value has surged or plummeted using an average value, standard deviation, quantile, or threshold value based on the time series distribution of data according to the repurchase pattern.

[0127] The processor (110) can generate insight data including the results of determining whether there is a surge and a fall.

[0128] Insight data can be utilized as real-time analytical indicators to support user decision-making and can be provided in a form that is immediately usable without the need for external analysis systems.

[0129] For example, the process of generating insight data can be performed by a processor (110) within an internal data space without calling an external system (10).

[0130] For example, insight data can be generated by the processor (110) without additional data movement. The insight data is provided to the user terminal (20) by the processor (110) to provide real-time or near-real-time analysis results simultaneously with data processing.

[0131] For example, a first model that has been trained to predict at least one purchase scenario for input target data and to calculate the probability of occurrence of the predicted purchase scenario may be stored in the memory (120). The first model may be called from the memory (120) by the processor (110) to perform a calculation process.

[0132] A processor (110) according to one embodiment of the present invention can train a first model to predict at least one purchase scenario and the probability of occurrence of the purchase scenario based on the target data, using the time-series order of identifiers included in the target data and the combination of identifiers based on the accumulated target data.

[0133] Meanwhile, the processor (110) can take the target data as input and predict the result of determining whether the target data will surge or fall based on the output result of the first model.

[0134] The processor (110) can provide the predicted judgment result to the user terminal (20).

[0135] The processor (110) can compare the insight data with the prediction results of the first model.

[0136] As a result of the comparison, if there is a discrepancy in whether there is a sharp rise or a sharp fall, or if the difference in the calculated degree of fluctuation exceeds a threshold, the processor (110) can store the prediction result and target data as noise data.

[0137] The processor (110) can analyze the number of noise occurrences based on the identifiers included in the accumulated stored noise data.

[0138] The processor (110) can identify at least one common identifier among the identifiers included in the accumulated stored noise data, wherein the number of noise occurrences is greater than or equal to a threshold number.

[0139] When a common identifier is identified, the processor (110) can calculate a first weight for the common identifier that is inversely proportional to the number of noise occurrences.

[0140] The processor (110) can store the calculated first weight corresponding to a common identifier.

[0141] When new target data including a common identifier is received, the processor (110) can apply a first weight to correct and adjust the probability of a purchase scenario occurring for the new target data of the first model in a decreasing direction.

[0142] Such weight-based correction can form a self-correcting mechanism that continuously improves prediction accuracy by learningly reflecting data outliers or recurring error patterns.

[0143] A processor (110) according to one embodiment of the present invention can classify multiple target data based on multiple occurrence addresses where data occurred.

[0144] For example, a plurality of occurrence addresses may include a user terminal (20) or an external system (10).

[0145] The processor (110) can calculate a purchase scenario and a purchase probability for each occurrence address based on a plurality of first models configured independently for each occurrence address.

[0146] The processor (110) can calculate the probability of purchase for each of multiple identifiers for each occurrence address-purchase item based on the actual purchase history for each occurrence address.

[0147] The processor (110) can calculate the average purchase probability of each of a plurality of identifiers for each common purchase item based on at least one common purchase item, which is a purchase item sold equally for a plurality of occurrence addresses.

[0148] The processor (110) can calculate the ranking of the purchase probability of multiple identifiers for each common purchase item.

[0149] The processor (110) can compare the difference between the purchase probability per occurrence address corresponding to the top identifier and the average purchase probability based on the top identifier that is above the threshold rank among the multiple identifiers.

[0150] The processor (110) can identify a first occurrence address corresponding to a purchase probability lower than the average purchase probability by more than a threshold as a result of comparison.

[0151] Additionally, the processor (110) can identify a second occurrence address corresponding to a purchase probability higher than the average purchase probability as a result of comparison.

[0152] The processor (110) can select at least one first purchase scenario including a high identifier for a first occurrence address.

[0153] The processor (110) can select at least one second purchase scenario including a higher identifier for the second occurrence address.

[0154] The processor (110) can take a first purchase scenario and a second purchase scenario as inputs and generate or train a third model that corrects the output result of a second model that has been trained for a first occurrence address among a plurality of first models.

[0155] For example, the third model may be an artificial intelligence model trained to calculate a correction value by reflecting the deviation by address for the probability of a purchase scenario generated as the output of the second model, using as training data the first purchase scenario, the second purchase scenario, the purchase probability by address for common purchase items, the average purchase probability and deviation value by address, and the correct answer data regarding whether there is a surge or a surge based on actual purchase history. The third model may be stored in memory (120). The third model may be called from memory (120) by the processor (110) to perform a computation process.

[0156] The processor (110) can apply the correction value calculated by the third model to the output of the second model to generate a corrected purchase scenario occurrence probability with the bias by occurrence address removed.

[0157] The processor (110) can recalculate the result of determining whether there is a surge or a drop in the common purchase item of the first occurrence address based on the probability of the corrected purchase scenario occurring.

[0158] This enables the elimination of address-specific bias that may occur in single-model-based predictions and the correction of statistical imbalances among various data sources.

[0159] FIG. 6 is an additional operation flowchart of a method according to another embodiment.

[0160] Meanwhile, referring to FIG. 6, the processor (110) can receive API specifications, data schema information and unstructured data structures (S610) from an external system (10).

[0161] The processor (110) can parse API specifications and data structures using a natural language processing and structure analysis-based interpretation module (Parsing, the process of structurally interpreting and breaking down data so that it can be written) (S620).

[0162] The processor (110) can extract structural elements (S630) including endpoint information, authentication method, request and response format, data field configuration, and relationship information between fields from the parsing result.

[0163] The processor (110) can generate a metadata schema (S640) that includes API call rules, data conversion rules, and processing flow definition information based on structural elements.

[0164] For example, a metadata schema may be a structured data set that includes at least a node identifier, input data type, output data type, processing function information, API call information, and dependency relationship information between nodes.

[0165] The processor (110) can store the metadata schema as a single integrated execution definition information (S650).

[0166] The processor (110) can call a metadata execution engine (S650) that generates an execution graph based on a metadata schema.

[0167] For example, the execution graph may be composed of a Directed Acyclic Graph (DAG) that includes data flow nodes, API call nodes, and conditional branch nodes.

[0168] Here, the execution graph defines execution logic as a metadata-based structure rather than in the form of code, which can be dynamically interpreted and configured at runtime.

[0169] Therefore, multiple nodes constituting the execution graph structure can be defined to form an acyclic graph structure that defines data flow and control flow by being connected according to the dependency relationship between the preceding node and the succeeding node.

[0170] In addition, the generated acyclic graph structure can be stored as static execution definition information that does not contain execution logic.

[0171] In addition, static execution definition information can be converted into a structure that can be interpreted by a metadata execution engine after storage and provided for use in execution.

[0172] For example, a metadata execution engine can dynamically generate multiple functional units, including data collection, data transformation, data filtering, and external API calls, based on an execution graph. The metadata execution engine executes the generated multiple functional units using an interpreter-based interpretation method, enabling the functional configuration to be dynamically reconfigured without changing the application source code.

[0173] Here, a functional unit may refer to a minimum executable unit module that performs at least one of a single API call, a data transformation function, or a conditional branch operation.

[0174] Accordingly, functional changes can be made without system redeployment, and the speed of response to external system changes can be improved.

[0175] For example, when a metadata execution engine receives function block definition information input through a user interface, it can interpret the function block definition information to identify multiple processing elements including API call steps, data transformation steps, and conditional branching conditions.

[0176] Accordingly, the metadata execution engine can generate node-level meta-definition information by analyzing the input data type, output data type, processing function definition, and inter-processing dependency relationships for each of the multiple processing elements.

[0177] The metadata execution engine can be configured to include data flow relationships and control flow relationships between each processing element in the node-level meta-definition information.

[0178] The metadata execution engine can generate multiple nodes, including API call nodes, data transformation nodes, conditional branching nodes, and data merging nodes, based on node-unit meta-definition information.

[0179] A processor (110) according to one embodiment of the present invention can receive a metadata schema.

[0180] The processor (110) can interpret API call rules, data transformation rules, and execution definition information included in the metadata schema and convert them into an execution graph structure including data flow relationships and control flow relationships.

[0181] The processor (110) can generate composite state information in real time for each of the multiple nodes constituting the execution graph structure, including the schema compatibility of the input data, the execution completion status of the preceding node, the external system response status, and whether an event trigger occurred.

[0182] The processor (110) can determine whether each node is executable based on composite state information using a state combination evaluation method.

[0183] For example, the state combination evaluation method may be a rule-based evaluation method that determines whether execution is possible by combining the validity of input data, whether the preceding node is completed, and the external system response state as logical operation conditions.

[0184] The processor (110) can dynamically load and execute functional units including API call functions, data conversion functions, or conditional branching functions only for executable nodes.

[0185] The processor (110) can dynamically calculate the execution order based on data dependency relationships and control flow dependency relationships between nodes.

[0186] For example, the execution order can be determined by the processor (110) as an adaptive execution order that is reordered according to changes in state.

[0187] When an external event occurs during execution according to the execution order, an API request occurs, an external system (10) response delay occurs, or input data changes occur, the processor (110) can detect a set of nodes corresponding to the range of influence without reconstructing the entire execution graph.

[0188] The processor (110) can generate a partial execution graph based on a set of detected nodes.

[0189] For example, a partial execution graph may be configured to include nodes within the scope of influence and trailing nodes that depend directly or indirectly on those nodes.

[0190] The processor (110) can update the execution path by recalculating dependency relationships and execution conditions for the partial execution graph.

[0191] The processor (110) can continue execution by reflecting the API call order, data conversion order, and conditional branch path in real-time without interruption according to the updated execution path.

[0192] This involves selectively updating only the parts affected by the change without reconstructing the entire execution graph, which can be a key feature enabling continuous service provision without system interruption.

[0193] Meanwhile, the memory (120) may store a fourth model, which is a graph-based impact prediction artificial intelligence model configured to output an impact range node set and an execution path change pattern when a node change occurs, by learning based on past changes in multiple external systems (10), metadata schema version change history, execution graph structure change history, and execution success and failure result data. The fourth model may be called from the memory (120) by the processor (110) to perform a computation process.

[0194] For example, the fourth model may be a model for predicting the range of influence on the execution graph when a node change occurs, and may be a model dedicated to influence prediction in which the output value is limited to the node set and propagation order, and no modification of the execution logic or change of the execution engine is performed.

[0195] A processor (110) according to one embodiment of the present invention can monitor whether the node structure of a metadata schema matching each of a plurality of external systems (10) has been changed.

[0196] The processor (110) can identify the first external system among the plurality of external systems (10) where a node structure change has occurred.

[0197] The processor (110) can determine that an update has occurred in the first external system based on the time when the first external system is identified.

[0198] When it is determined that an update has occurred in the first external system, the processor (110) can obtain the first metadata schema generated after the update in correspondence with the first external system.

[0199] Additionally, the processor (110) can obtain a second metadata schema created prior to the update in correspondence with the first external system.

[0200] The processor (110) can classify the first segments where the data flow is the same by comparing the first metadata schema and the second metadata schema.

[0201] The processor (110) can classify a second section with a different data flow by comparing the first metadata schema and the second metadata schema.

[0202] The processor (110) can identify a common node, which is a node connecting the first section and the second section.

[0203] Additionally, the processor (110) can identify a plurality of change nodes that are nodes included in the second section.

[0204] The processor (110) can generate change composite state data by collecting state information including node structure information, data type change history, API call failure history, execution order change history, and external system response delay history for a common node and a plurality of change nodes.

[0205] For example, change composite state data can be configured in the form of a vector including a node identifier, change type, data type change information, call success status, and response delay time information.

[0206] The processor (110) can predict the range of influence and propagation order of a node change on the entire execution graph through a fourth model using change composite state data as input.

[0207] For example, the scope of influence is defined as the set of nodes propagated from the change node, and the propagation order may refer to the execution order based on dependencies between nodes.

[0208] Here, the fourth model is a segregated analysis model that predicts only the range of influence without directly modifying the execution path, and can operate independently of the execution system.

[0209] The processor (110) can predict the set of nodes included in the update influence range of the first external system based on the output result of the fourth model.

[0210] The processor (110) can generate a predicted execution path on an execution graph based on a predicted set of nodes.

[0211] Meanwhile, the memory (120) may store a fifth model, which is a graph-based execution engine correction artificial intelligence model configured to predict execution stability and processing accuracy when correction logic is applied and output a corrected execution engine, by learning based on past execution engine correction history, node-unit correction success and failure history, API call stability history, and execution graph structure change history. The fifth model may be called from the memory (120) by the processor (110) to perform a computation process.

[0212] For example, the fifth model is a model for optimizing changes in the operation of an execution engine resulting from structural changes in the execution graph or the application of correction logic, and may be a model that reconstructs the execution engine itself based on structural compatibility and execution stability indicators between the correction logic and the execution engine, without directly using a set of influence range nodes or execution path prediction results as input.

[0213] A processor (110) according to one embodiment of the present invention can determine whether it is necessary to generate correction logic for each of a plurality of change nodes based on the degree of influence that the change node has on the execution logic of the metadata execution engine.

[0214] The processor (110) can generate correction logic that reflects the type of change of data conversion rules, API call rules, or conditional branch rules for a change node that requires correction logic generation.

[0215] For example, the correction logic may be a set of rules including at least one of modifying a data conversion function, changing API call parameters, or changing a conditional branching condition.

[0216] The processor (110) can identify at least one second external system that includes an execution graph having a structural similarity with the correction logic greater than or equal to a threshold among metadata execution engines that match each of the plurality of external systems (10) in conjunction with the generated correction logic.

[0217] The processor (110) can generate a candidate execution engine to be corrected by replicating the metadata execution engine of the second external system.

[0218] The processor (110) can generate a corrected execution engine through the fifth model by using the correction logic, the duplicated correction target execution engine candidate, and the metadata execution engine of the first external system as integrated inputs. This allows the correction to be performed by reconstructing the execution engine itself rather than modifying a part of the execution logic, thereby enabling the maintenance of a stable execution environment even with complex structural changes.

[0219] Here, the input to the fifth model may consist of a feature vector including correction logic information, execution engine structure information, and past execution success / failure history.

[0220] When the data size of the first metadata schema is less than the threshold capacity, the processor (110) can generate test data by replicating the data flow and branch path between nodes based on the execution graph structure of the first metadata schema.

[0221] The processor (110) can generate test execution data by applying a corrected execution engine to the test data.

[0222] The processor (110) can calculate an engine score including execution accuracy and processing stability based on test execution data.

[0223] For example, the engine score can be calculated based on execution success rate, average response time, and error frequency.

[0224] The processor (110) can generate a corrected engine score by calculating a second weight based on the ratio of the test data size and the metadata schema size and applying it to the engine score.

[0225] If the corrected engine score exceeds the threshold score, the processor (110) can update and apply the metadata execution engine of the first external system to the corrected execution engine.

[0226] This pre-verification-based application method can prevent the possibility of execution errors in advance and improve stability in actual operating environments.

[0227] A device according to one embodiment of the present invention may be combined with hardware and controlled by a computer program stored in a medium to execute any one of the operation methods by the processor (110) of the system described above.

[0228] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0229] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0230] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0231] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0232] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 In a device providing a data refinement and processing solution that improves the processing speed of large-capacity data, the device comprises: a memory; a communication unit; The system includes one or more processors connected to the memory and communication unit, wherein the processor receives target data including large-capacity structured data and unstructured data from a user terminal or an external system, loads the target data into an internal data space within the cloud infrastructure without passing it to the application layer, and is configured as an in-memory execution environment based on a single address space where a storage area for data storage and a computation area for data computation are not logically separated, and based on the data loaded into the internal data space, generates an index structure by analyzing the data structure in real time without undergoing serialization and deserialization processes, and databases the data without calling an external database, and for the databased data, directly executes data processing logic including cleaning, filtering, aggregation, arithmetic operations, and statistical operations within the internal data space without performing network-based data transmission and reception with an external application server or an external database server, and the entire process from receiving the target data to database creation and data processing logic is performed within a single execution environment, and the internal data space is composed of multiple physical nodes, each node including independent local memory while logically sharing a single address space, distributed It includes a distributed shared memory structure, and the processor analyzes the structure, size, and memory access patterns of data loaded in the internal data space and divides the data processing task into a plurality of task units generated in units of data blocks or memory pages.A device for providing a data cleansing and processing solution that improves the processing speed of large-capacity data, characterized by scheduling the plurality of work units to minimize local memory access by considering the cache structure and NUMA (Non-Uniform Memory Access) characteristics of each CPU core, executing the plurality of work units in parallel on an in-memory-based parallel processing pipeline, and, while executing the plurality of work units in parallel, performing data exchange between nodes using a memory-mapping-based shared memory access method without using network protocol-based communication, thereby restricting data movement to occur only within the memory layer. Claim 2 delete Claim 3 A device for providing a data refinement and processing solution that improves the processing speed of large-capacity data, characterized in that, in claim 1, the processor connects multiple transaction data based on the same identifier based on the data processing result by data processing logic within the internal data space, analyzes a repurchase pattern including data sorting in chronological order, identifier-based grouping, and determination of whether to make repeated purchases based on the connection structure of the multiple transaction data, determines whether the data value surges or plummets using an average value, standard deviation, quantile, or threshold value based on the time-series distribution of the data according to the repurchase pattern, and generates insight data including the result of determining whether to surge or plummet, wherein the process of generating the insight data is performed within the internal data space without calling an external system, and the insight data is generated without additional data movement and provided to a user terminal, thereby providing real-time or near-real-time analysis results simultaneously with data processing.

Images