A data-driven data processing method, system, medium and program product

Abstract

The application discloses a data piece driven data processing method, system, medium and program product, relates to the field of electric digital data processing, and the method comprises the following steps: acquiring source data, encapsulating the source data into an executable data unit, and generating a data piece; mapping the data piece into a data semantic coordinate, and constructing a data index tree of a unique semantic identifier, a data storage address and the data semantic coordinate; converting a data request into a target semantic coordinate, and determining a target semantic identifier and a target data piece; constructing a transmission connection with a target storage node with the minimum delay of the target data piece; splitting the target data piece into a logical control segment containing a standardized operation interface and one or more data entity segments; transmitting the logical control segment to a requester, so that the requester starts the standardized operation interface before receiving the data entity segment; and pulling and synchronously transmitting the corresponding data entity segment from the target storage node. By implementing the application, the data circulation cost between different systems can be reduced.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing, and in particular to a data processing method, system, medium, and program product driven by a data device. Background Technology

[0002] With the development of the digital economy, data has become an important factor of production, and the integration and efficient utilization of multi-source heterogeneous data has become a core requirement for data-intensive applications. In application scenarios such as big data and artificial intelligence, it is necessary to manage, quickly retrieve, and efficiently transmit massive amounts of data from different sources, formats, or semantics to support the flow and application of data between different systems and entities.

[0003] In related technologies, Uniform Resource Locators (URLs) are generally used as unique identifiers for data resources, and data is retrieved and transmitted over the network via HTTP / HTTPS protocols. For data management, centralized storage solutions such as data warehouses or data lakes are used to integrate multi-source data, and heterogeneous data is converted into a unified format through an ETL (Extract-Transform-Load) process before storage. During data retrieval, index queries are performed based on URL addresses and metadata directories; users need to explicitly know the data's storage location and access path. During data transmission, the client initiates a request to the server, establishes a connection, and then synchronously downloads the data. Users must wait for the data to be completely downloaded before they can begin using it.

[0004] However, the data management methods in related technologies result in data resources being tightly coupled with specific application systems. Data transfer between different systems requires format conversion and interface adaptation, which increases the cost of data circulation. Summary of the Invention

[0005] This application provides a data processing method, system, medium, and program product driven by data components, which can reduce the cost of data flow between different systems.

[0006] Firstly, this application provides a data component-driven data processing method applied to a data processing system. The method includes: acquiring source data; encapsulating the source data and corresponding standardized operation interfaces into executable data units based on a standardized operation protocol to generate data components with unique semantic identifiers; mapping the data components to data semantic coordinates in domain, time, and geographic dimensions, and constructing a data index tree containing unique semantic identifiers, data storage addresses, and data semantic coordinates; receiving data requests; converting the semantic information of the data requests into target semantic coordinates and retrieving them in the data index tree to determine the target semantic identifier and target data component; determining multiple storage nodes for the target data component and establishing a transmission connection with the target storage node with the lowest latency; splitting the target data component into a logical control segment containing a standardized operation interface and one or more data entity segments according to the data structure logic; transmitting the logical control segment to the requester, enabling the requester to start the standardized operation interface before receiving the data entity segments; and during the operation of the standardized operation interface, pulling and synchronously transmitting the corresponding data entity segments from the target storage node according to external access instructions.

[0007] In the above embodiments, the data processing system decouples data from the application system by encapsulating data and operation interfaces into data components and using semantic coordinates for indexing. By transmitting logical control segments first and then pulling data entity segments on demand, the requester can start processing without waiting for all data to be downloaded, reducing the data flow cost between different systems and improving data utilization efficiency.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the logical control segment includes a standardized operation interface and a data segment dependency graph describing the relationships between data entity segments; after the step of pulling and synchronously transmitting the corresponding data entity segment from the target storage node according to the external access instruction during the operation of the standardized operation interface, the method further includes: when the standardized operation interface receives an external access instruction pointing to the first entity segment, querying the data segment dependency graph in the logical control segment to obtain a second entity segment that has a dependency relationship with the first entity segment; generating a transmission task list containing the first entity segment and the second entity segment, and setting the transmission priority of the second entity segment to be higher than that of other unrequested entity segments; and initiating a data pull request for the first entity segment and the second entity segment to the target storage node according to the transmission task list.

[0009] In the above embodiments, the data processing system utilizes a data fragment dependency graph to predict the data that the requester may need later when requesting a specific data fragment, and prefetches it. This optimizes multiple serial requests that may have occurred into a single parallel request that includes primary and secondary targets, reducing the number of request interactions and the total waiting time caused by accumulated network latency, thereby improving the continuity of data access and transmission efficiency.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of transmitting the logical control fragment to the requester so that the requester starts the standardized operation interface before receiving the data entity fragment, the method further includes: sending an instantiation instruction to the requester so that the requester generates an interface proxy object locally based on the interface definition in the logical control fragment; when the requester's application calls any method in the interface proxy object, the interface proxy object performs a check operation to determine whether the required entity fragment already exists in the local cache; if the check result is yes, the requester's call request and call parameters are forwarded to the loaded standardized operation interface, and the execution result of the standardized operation interface is returned; if the check result is no, an asynchronous response instruction is sent to the requester so that the requester generates a preset result object and returns it to the application; a data retrieval request containing the required entity fragment is sent to the target storage node, the returned data entity content is received, and the preset result object is adjusted with the data entity content.

[0011] In the above embodiments, the data processing system shields the upper-layer application from the complexity of acquiring underlying data by introducing an interface proxy object on the requesting side. This proxy object, combined with local caching and asynchronous loading mechanisms, ensures that the application's calls are not blocked due to data not being ready, but instead obtain a usable result object. This object is populated after the background data arrives, improving the application's response speed and the smoothness of the user experience.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, the data request includes joint computation logic for multiple target data items; after determining multiple storage nodes for the target data items and establishing a transmission connection with the target storage node with the lowest latency, the method further includes: decomposing the joint computation logic into one or more sub-computation logics that can be executed independently on a single data item and an aggregation logic; matching each target data item with a computation instruction containing the corresponding sub-computation logic and computation parameters; sending the computation instruction to the target storage node hosting each target data item and triggering the standardized operation interface of the corresponding target data item to execute the corresponding sub-computation logic on the target storage node and generate intermediate computation results; collecting all intermediate computation results from each target storage node and executing the aggregation logic on its own computation node or a preset computation node to obtain the final computation result and sending it to the requester.

[0013] In the above embodiments, the data processing system decomposes the joint computing logic and pushes it down to the storage node where the data is located for execution, thereby realizing the movement of computing to data. This avoids transmitting multiple large-scale raw data files distributed on different nodes to a single computing node, reducing the amount of data transmitted over the network, and reducing network bandwidth pressure and data transmission latency.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of decomposing the joint computing logic into one or more sub-computing logics that can be executed independently on a single data element and an aggregation logic, the method further includes: querying the metadata of a single data element to obtain an estimated data size of each data element and the available computing resources of the target storage node where it is located; calculating the estimated execution cost of each sub-computing logic based on the complexity of the sub-computing logic, the estimated data size, and the available computing resources; determining the corresponding sub-computing logic as complex computing logic when the estimated execution cost exceeds a preset cost threshold; and pulling the data entity fragments required to execute the complex computing logic from the corresponding target storage node to a designated computing node to execute the complex computing logic on the designated computing node.

[0015] In the above embodiments, the data processing system introduces a cost-based computing task scheduling strategy. Before pushing down the computation, the system evaluates the execution cost of each subtask on the storage node. For complex computations that exceed the node's processing capacity, the system pulls the required data to a dedicated computing node for execution. This avoids the problem of storage node overload or low computation efficiency caused by forcibly pushing down complex tasks, and achieves a reasonable allocation of computing load among nodes with different capabilities.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, after determining that the corresponding sub-computation logic is complex computation logic when the estimated execution cost exceeds a preset cost threshold, the method further includes: decomposing the complex computation logic into local preprocessing logic and centralized aggregation logic; the local preprocessing logic includes filtering operation logic or transformation operation logic to reduce the data size; sending the computation instructions containing the local preprocessing logic to the corresponding target storage node to generate lightweight intermediate data with a data size smaller than the corresponding data entity fragment on the target storage node; and pulling the lightweight intermediate data from the target storage node to a designated computing node to execute the centralized aggregation logic on the designated computing node to obtain the final execution result.

[0017] In the above embodiments, the data processing system generates lightweight intermediate data by performing data reduction preprocessing on the storage node, and then transmits the intermediate data to the computing node for final aggregation. In the case that data transmission cannot be completely avoided, the amount of data transmitted is minimized.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after determining the addresses of multiple storage nodes of the target data file and the network latency parameters of each node, and establishing a transmission connection with the node address with the lowest latency, the method further includes: reading the metadata of the target data file, extracting the source data hash value on which the metadata depends, and the algorithm logic and algorithm parameters for performing preprocessing and vector generation processing; sending a comparison instruction containing the source data hash value to the requester; when it is determined from the matching signal returned by the requester that the requester has basic data with the same source data hash value, prohibiting the transmission of data entity fragments in the target data file; generating a reconstruction instruction package containing a standardized operation interface, algorithm logic, and algorithm parameters, and sending the reconstruction instruction package to the requester, so that the requester generates the corresponding data entity fragment locally based on the basic data and the reconstruction instruction package.

[0019] In the above embodiments, for data components derived from general basic data, the data processing system determines whether the requester has the basic data required to generate the data component by comparing the hash value of the source data. If so, only the generation logic is transmitted instead of the data entity itself, and the requester completes the reconstruction locally, avoiding the network transmission of redundant data and saving a lot of bandwidth when processing derived datasets.

[0020] In a second aspect, embodiments of this application provide a data processing system comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the data processing system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a data processing system, cause the data processing system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a data processing system, cause the data processing system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the data processing system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By employing a technical solution that encapsulates source data and standardized operation interfaces into executable data units to generate data components, uses data semantic coordinates to build an index tree for retrieval, and splits data components into logical control fragments and data entity fragments with priority given to transmitting logical control fragments, data is decoupled from specific applications and storage locations. This enables content- and semantic-based data discovery, and the requesting party can initiate data processing logic after receiving a small amount of control information without waiting for the complete transmission of massive data entities. This achieves low-cost, high-efficiency flow and instant access to cross-system data, enhancing the mobility and utilization value of data as a production factor.

[0026] 2. By employing a technical solution that includes a data segment dependency graph describing the relationships between data entity segments within the logical control segment, and querying the dependency graph to obtain and prioritize the transmission of second entity segments with dependencies upon receiving an access instruction for the first entity segment, the data processing system can predict the contextual relevance of data requests and preload data segments that are likely to be requested subsequently. This achieves intelligent data prefetching, reduces the number of request interactions and the overall waiting time, and provides users with a smoother and more efficient data interaction experience.

[0027] 3. By adopting a technical solution that decomposes the joint computation logic across data components into sub-computation logic and aggregation logic that can be executed independently on each data component, and pushes the sub-computation logic down to the storage nodes where each target data component resides, and performs computation locally using its standardized operation interface, and finally only collects the intermediate aggregation results, large-scale data processing tasks are distributed to the data source for completion nearby, avoiding the need to migrate multiple original datasets as a whole to the central computing node; this realizes a distributed computing paradigm that moves computation to data, significantly reduces data transmission overhead, and improves the response speed of joint analysis tasks and the overall throughput of the system. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a data processing method driven by data components in an embodiment of this application.

[0029] Figure 2 This is another flowchart illustrating the data processing method driven by data components in the embodiments of this application;

[0030] Figure 3 This is a schematic diagram of the physical device structure of a data processing system in an embodiment of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] In the specific implementation scenarios of this application, there are several technical terms. For example, in a smart city traffic management platform, traffic data from different areas (such as A1 area and A2 area), different devices (such as geomagnetic sensors and video surveillance), and different times (such as morning peak and off-peak) are collected.

[0034] In the technical solution of this application, a traffic flow statistics table for the morning rush hour on Century Avenue, District A1, City A, on December 20, 2025, can be encapsulated into a data component. This data component not only contains tabular data of traffic flow (i.e., source data) but also embeds a standardized operation interface that provides methods such as getAverageSpeed() or getCongestionEvents(level). This data component is assigned a globally unique semantic identifier (e.g., a UUID) and mapped to a data semantic coordinate system, where the coordinates can be {domain: traffic, geography: City A / District A1 / Century Avenue, time: 2025-12-20}. (It should be noted that specific terms such as date and region in this document are for illustrative purposes only and should not be construed as limiting the scope of the data.)

[0035] When a traffic analysis application needs this data, it doesn't request a specific URL, but rather initiates a semantic data request, such as querying the morning rush hour traffic data for Century Avenue in Zone A1. The data processing system parses this request and locates the target data element using a data index tree. During transmission, the system first sends a logical control fragment containing interface definitions such as getAverageSpeed(). The analysis application can then display available analysis options on its interface. When the user clicks to calculate the average speed, the application issues an external access command, and the system then retrieves the corresponding data entity fragment from the nearest target storage node (such as an edge server deployed in City A) to complete the calculation. This process embodies the end-to-end innovation of this solution, from data encapsulation and retrieval to transmission, achieving efficient and semantic data flow.

[0036] The following describes the process of the method provided in this implementation. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a data processing method driven by data components in an embodiment of this application.

[0037] S101. Obtain source data, and encapsulate the source data and corresponding standardized operation interface into executable data units based on standardized operation protocols to generate data components with unique semantic identifiers.

[0038] In this context, source data refers to raw, unencapsulated data of various types, such as database tables, text files, images, and videos. A standardized operation protocol is a predefined set of rules used to regulate how data is accessed and manipulated. A standardized operation interface is a set of functions or methods implemented according to this protocol and bound to the source data. An executable data unit refers to an encapsulated entity that possesses self-describing and interactive capabilities. A unique semantic identifier is a globally unique ID assigned to each data element for precise referencing within the system. A data element is a collection of source data, standardized operation interfaces, and metadata, and is the basic unit of data circulation.

[0039] Specifically, this step is triggered when the data processing system needs to incorporate new data into its management. The system first reads the content and structure of the source data, and then selects or generates a suitable set of standardized operation interfaces based on a preset template or analysis of the data content. For example, for tabular data, an interface supporting row-by-row and column-by-column queries can be generated; for time-series data, an interface supporting slicing by time range can be generated. Subsequently, the data processing system packages the source data and the compiled interface code (or interface definition) into a logical unit, i.e., a data component, and assigns it a unique semantic identifier generated through a hash algorithm or UUID mechanism.

[0040] It's important to note that during data analysis, the data processing system initiates a data probing module. This module uses a series of heuristic rules and statistical analyses to identify the basic structure and type of the data. For example, it checks file extensions (e.g., .csv, .json, .parquet), reads the file header bytes to identify binary formats (e.g., video, audio), and samples text data. For the sampled data, the system applies regular expression matching to determine field types, such as identifying strings conforming to the ISO 8601 standard as time-series data, identifying fields containing longitude, latitude, or WKT (Well-Known Text) format as geospatial data, and statistically analyzing the distribution of numerical fields to determine whether they are discrete or continuous. After completing the data profiling, the system generates standardized operation interfaces based on a pre-defined data feature-interface template mapping rule base. For instance, if the system analyzes a file and finds it to be a multi-column structure, with one column identified as a timestamp and another as a numerical value, the system will determine it to be time-series data and apply a time-series data interface template to generate interfaces such as:

[0041] getTimeRange(startTime, endTime),

[0042] Methods such as getLatestValue() and getAverageByInterval(interval) are used, and these interface definitions and implementation logic are encapsulated into data components.

[0043] In some embodiments, data components can be generated in several ways: Optionally, containerization technology can be used, where the source data is treated as a data volume, and the standardized operation interface is implemented as a lightweight web service, all packaged together in a container image, which is the data component; alternatively, an archive file format can be used, such as designing a special ZIP or TAR package structure, which includes a manifest file describing metadata and interface definitions, a data file storing the source data, and a script file (such as WASM or JavaScript) containing the interface implementation logic, all three constituting the data component. It is understood that other methods can also be used to encapsulate the data component, such as storing it in a computation-enabled object storage system, which is not limited here.

[0044] In some embodiments, there may be security issues with standardized operation interfaces, meaning the interface code may contain malicious operations. To address this, the data processing system performs static code scanning and dynamic sandbox testing when encapsulating interfaces. Static scanning is used to discover known code vulnerabilities and dangerous function calls. Dynamic sandbox testing runs the interface methods in an isolated environment, monitoring their system calls, network access, and file read / write behavior to ensure that their behavior conforms to preset security policies. Only interfaces that pass the security verification are encapsulated into data components.

[0045] S102. Map the data components to data semantic coordinates in the domain dimension, time dimension, and geographic dimension, and construct a data index tree with unique semantic identifiers, data storage addresses, and data semantic coordinates.

[0046] In this context, semantic coordinates refer to a multi-dimensional descriptive system used to characterize the content of data items from business, time, and spatial perspectives. The domain dimension represents the industry or theme to which the data belongs, such as finance or healthcare. The time dimension represents the point in time or time period associated with the data. The geographic dimension represents the spatial location associated with the data. The data storage address refers to the physical or logical location where the data item is stored, which may be one or more URLs. A data index tree is an efficient data structure, such as a B+ tree or Radix tree, used to store and retrieve the mapping relationship between semantic coordinates and data item identifiers and addresses.

[0047] Specifically, after generating the data file in step S101, the data processing system parses the metadata of the data file (which may be provided by the data provider or extracted by the system). The metadata contains key information describing the data content, and the system generates corresponding semantic coordinates for the data file based on this information. For example, the semantic coordinates of a data file about the air quality report of City A in the third quarter of 2025 can be determined as {domain: environment, geography: City A, time: 2025-Q3}. Then, the data processing system inserts this semantic coordinate, the unique semantic identifier of the data file, and its address on one or more storage nodes as an index record into the global data index tree.

[0048] In some embodiments, the index tree can be constructed and maintained in several ways: Optionally, a distributed hash table (DHT) can be used to construct the index, combining different dimensions of semantic coordinates and hashing them, distributing the index records across multiple nodes in the network to achieve a decentralized index service; alternatively, a centralized multidimensional index database can be used, such as a database that supports geospatial and time-series indexing (e.g., PostGIS, TimescaleDB), to build a high-performance central index service. It is understood that other methods can also be used to implement the index structure to support efficient semantic retrieval, and these are not limited here.

[0049] It's important to note that dimensional mapping relies on a built-in semantic annotation engine that combines an ontology library and rule-based reasoning. First, the system extracts all noun phrases and keywords from the metadata of data items (such as filenames and descriptions provided by the data provider) and sampled content of data entity fragments. Then, the system matches these extracted terms with a multi-dimensional ontology library. This ontology library predefines concept hierarchies and synonyms across three dimensions: domain, geography, and time. For example, the geography dimension ontology library contains an administrative division tree from country and province to city and street; the domain dimension ontology library contains industry terms such as finance, transportation, and meteorology, and their hypotheses (such as stocks, traffic flow, and rainfall). When the metadata of a data item contains "City A's GDP report for the first quarter of 2025," the semantic annotation engine identifies City A by matching the geography ontology library, identifies the first quarter of 2025 by matching the time ontology library, converts it to a standard time range [2025-01-01, 2025-03-31], and identifies GDP as belonging to the economic domain by matching the domain ontology library. Finally, the system synthesizes these matching results to generate structured semantic coordinates for the data item {domain: economy, geography: city A, time: [2025-01-01, 2025-03-31]}. For ambiguous words, the system will combine contextual information or assign different confidence scores to multiple possible coordinates.

[0050] S103. Receive a data request, convert the semantic information of the data request into target semantic coordinates, and search in the data index tree to determine the target semantic identifier and target data file.

[0051] In this context, a data request refers to a query initiated by a requester (such as an application or another service) that describes its intended data request. Semantic information refers to the unstructured or semi-structured descriptive language contained in the request, such as a natural language query to find all sales reports for City A last year. Target semantic coordinates refer to the structured query conditions obtained after parsing the semantic information, which conform to the coordinate system of the data index tree. The target data file is the data file that matches the target semantic coordinates.

[0052] Specifically, this step is executed when the front-end service of the data processing system receives a data request. The Natural Language Processing (NLP) module or semantic parsing engine within the data processing system analyzes the request content, extracting key entities such as time, location, and topic. These entities are then mapped to dimensions in the data index tree, forming one or a set of target semantic coordinates. For example, a request to find all sales reports for City A last year is converted into target semantic coordinates {domain: sales, geography: City A, time: 2024-01-01 / 2024-12-31}. The data processing system uses these coordinates to perform range queries or exact matches in the data index tree, identifying the unique semantic identifiers of all data items that meet the criteria.

[0053] In some embodiments, request transformation and retrieval can be achieved in several ways: Optionally, by constructing a query parsing syntax tree, complex natural language requests can be decomposed into logical combinations (AND, OR, NOT) of multiple sub-query conditions, and then each sub-query condition can be converted into semantic coordinates for combined retrieval; alternatively, by training a sequence-to-sequence (Seq2Seq) machine learning model, the input natural language request text can be directly converted end-to-end into a structured target semantic coordinate query statement. It is understood that other methods can also be used to achieve semantic understanding and matching, which are not limited here.

[0054] In some embodiments, there may be situations where the number of search results is too large or there are no exact matches. To address this, the data processing system implements a hierarchical search and fuzzy matching strategy. If there are too many exact matches, the system will prompt the requester to add more specific criteria, or return the top N most relevant results based on relevance scores. If there are no exact matches, the system will broaden the query criteria, for example, expanding the geographical scope from District A1 of City A to City A, or applying fuzziness in the time dimension, and return these approximate matches for the requester to choose from.

[0055] S104. Determine multiple storage nodes for the target data file and establish a transmission connection with the target storage node with the lowest latency.

[0056] In this context, multiple storage nodes refer to the possibility that the same data file may be replicated and stored on different physical servers or data centers to achieve disaster recovery and proximity access. The target storage node with the lowest latency is the one with the lowest network latency among all nodes storing the data file. The transport connection refers to the network channel established between the requester and the target storage node for data exchange, such as a TCP connection.

[0057] Specifically, after determining the unique semantic identifier of the target data file in step S103, the data processing system queries the data index tree or a dedicated replica management service again to obtain a list of addresses of all storage nodes associated with that identifier. Next, the data processing system (or, under the guidance of the requester) sends a lightweight network probe packet (such as ICMPPing or a small TCP / UDP packet) to each storage node in the list and measures their respective round-trip time (RTT). By comparing the RTTs of all nodes, the node with the lowest latency is determined as the optimal source for this data transmission, i.e., the target storage node. Finally, the requester initiates a connection establishment process with the target storage node to prepare for subsequent data retrieval.

[0058] In some embodiments, the optimal node selection can be achieved in several ways: Optionally, a network detection module can be built into the requesting client, allowing the client to detect all candidate nodes and select the optimal node for connection, thus reducing the burden on the central server; alternatively, a global network topology and latency monitoring system can be maintained by the data processing system. This system periodically updates the network status between data centers, and when a request is received, the system directly calculates the optimal node based on the requester's IP address and the monitoring data, and returns it to the requester. It is understood that other methods can also be used to select nodes, such as comprehensively considering multiple factors such as network latency, node load, and bandwidth costs, which are not limited here.

[0059] S105. Based on the data structure logic, the target data component is split into a logical control segment containing a standardized operation interface and one or more data entity segments.

[0060] In this context, data structure logic refers to the organization of source data within a data file. For example, a CSV file consists of a header and data rows, while a video file consists of metadata, keyframes, and non-keyframes. Logical control segments are non-core data within a data file that possess control and descriptive capabilities; their core lies in the definition and implementation of standardized operation interfaces. Data entity segments are independently transmittable data blocks that constitute the main body of the source data.

[0061] Specifically, this step is executed on the target storage node, triggered when it receives a connection request from the requester. The target storage node performs splitting logic based on the type and internal structure of the target data file. For example, for a data file encapsulating a large CSV file, the node packages the file header (containing column definitions) and standardized operation interfaces (such as getRow(n)) into logical control fragments. Each line or several hundred lines of data in the file are treated as an independent data entity fragment. For a video data file, the video's metadata, encoding information, and operation interfaces (such as seekTo(time)) constitute the logical control fragment, while each GOP (Group of Pictures, video codec) can be considered a data entity fragment.

[0062] In some embodiments, data components can be split in several ways: Optionally, pre-splitting can be performed when the data component is created, storing the logical control segment and all data entity segments as independent objects in the object storage system, and recording the index relationship between them in the data component's metadata; alternatively, dynamic splitting can be performed during transmission, where the target storage node loads the entire data component in memory and dynamically generates and sends different segments according to requests. This approach is more flexible but places certain demands on node memory. It is understood that other methods can also be used to achieve splitting, the key being the separation of control logic from data entities; these are not limited here.

[0063] In some embodiments, data entity fragments may vary in size, leading to uneven transmission efficiency. To address this, the data processing system implements a block-based load balancing strategy when splitting data entities. The system sets a standard data entity fragment size (e.g., 1MB). For excessively large logical data units (such as a very long video GOP), it further divides them into multiple physical transport blocks of approximately the standard size. For numerous small, fragmented data units, multiple logical units are merged into a single physical transport block to reduce transmission protocol overhead and improve network utilization.

[0064] S106. Transmit the logic control segment to the requester, so that the requester can start the standardized operation interface before receiving the data entity segment.

[0065] In this context, starting the standardized operation interface refers to the requester loading and instantiating the interface defined in the logic control segment, putting it into a callable ready state, even if the complete data supporting the operation of the interface has not yet arrived.

[0066] Specifically, after the splitting is completed in step S105, the target storage node first sends a smaller logical control fragment to the requester through the established transmission connection. Upon receiving the fragment, the requester parses the interface definition and creates a proxy object in its local memory or loads the interface's execution environment (such as a WASM virtual machine). At this point, the application can already see and interact with the proxy object of this interface; for example, an interface for data analysis software can dynamically generate operable buttons and menus based on the interface definition. This process occurs before any large data entity fragment begins transmission.

[0067] In some embodiments, interface pre-startup can be implemented in several ways: Optionally, the logical control fragment contains a complete definition of the interface (such as an OpenAPI / Swagger definition file), and the requester generates a local stub or proxy class based on this definition. The methods of this proxy class are responsible for triggering subsequent data fetching when they are called. Optionally, the logical control fragment contains a lightweight, executable interface implementation (such as a WASM module), which is loaded into a sandbox by the requester and runs there. The module itself is responsible for managing the request and composition of data entity fragments. It is understood that other methods can also be used to implement interface pre-startup to achieve the effect of separation of control and data and immediate response, which are not limited here.

[0068] In some embodiments, the requesting environment may not support the interface technology stack in the logical control fragment (e.g., the browser does not support certain WASM features). To address this, the data processing system can provide logical control fragments in multiple formats when encapsulating data. Before transmission, the requesting party declares its supported environments to the target storage node (through mechanisms such as the HTTP Accept header). The target storage node then selects the most compatible logical control fragment version for transmission based on this declaration, for example, providing a WASM version for modern browsers and a JavaScript fallback version for older environments.

[0069] S107. During the operation of the standardized operation interface, the corresponding data entity fragments are retrieved from the target storage node and transmitted synchronously according to the external access command.

[0070] Here, an external access instruction refers to a call from the requesting application to a specific method in a standardized operation interface that has been started, such as calling getRows(100, 200). Synchronous transmission here refers to a transmission process initiated to satisfy the execution of the current instruction; once the data arrives, it is used to complete the calculation of the instruction, which is the opposite of asynchronous prefetching.

[0071] Specifically, when the requesting application calls a method of the interface proxy object instantiated in step S106, the proxy object (or its underlying runtime) determines which data entity fragments are needed to execute that method. For example, executing `getRows(100, 200)` requires the entity fragment containing rows 100 to 200. Subsequently, the proxy object initiates a pull request for the specific data entity fragment to the target storage node. Upon receiving the request, the target storage node reads the corresponding data entity fragment from its storage and sends it to the requesting party via a transport connection. Upon receiving the fragment, the requesting party places it into the runtime environment of the standardized operation interface, completes the initial method call, and returns the result to the application.

[0072] In some embodiments, on-demand data retrieval can be implemented in several ways: Optionally, an HTTPRange-Request-based approach can be used, treating the data file as a large file, with data entity fragments divided into byte ranges, and the requester requesting a specific byte range of data via the HTTP Range header; alternatively, a message queue-based protocol can be used, where each data entity fragment is treated as a message, the requester subscribes to the message IDs it needs, and the target storage node acts as the publisher, pushing data on demand. It is understood that other methods can be used to implement on-demand data retrieval, the core of which is to implement lazy loading for data access; these are not limited here.

[0073] In some embodiments, network jitter can cause data entity fragment transmission interruptions or excessive latency. To address this, the data processing system implements a fetch mechanism with breakpoint resumption and multi-source transmission. If a fragment's transmission is interrupted, the requester records the received portion and re-initiates the request from the point of interruption. Furthermore, if a data entity has multiple replica storage nodes, when the primary transmission link experiences a performance bottleneck, the requester can simultaneously fetch different parts of the same data entity fragment from other storage nodes and then concatenate them locally to improve transmission robustness and speed.

[0074] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the data processing method driven by data components in this application.

[0075] S201. Obtain source data, and encapsulate the source data and corresponding standardized operation interface into executable data units based on standardized operation protocols to generate data components with unique semantic identifiers.

[0076] Refer to step S101, which will not be repeated here.

[0077] S202. Map the data components to data semantic coordinates in the domain dimension, time dimension, and geographic dimension, and construct a data index tree with unique semantic identifiers, data storage addresses, and data semantic coordinates.

[0078] Refer to step S102, which will not be repeated here.

[0079] S203. Receive a data request, convert the semantic information of the data request into target semantic coordinates, and search in the data index tree to determine the target semantic identifier and target data file.

[0080] Refer to step S103, which will not be repeated here.

[0081] S204. Identify multiple storage nodes for the target data file and establish a transmission connection with the target storage node with the lowest latency.

[0082] Refer to step S104, which will not be repeated here.

[0083] S205. Decompose the joint computation logic into one or more sub-computation logics that can be executed independently on a single data element and an aggregate logic.

[0084] In this context, joint computation logic refers to computational tasks that require traversing multiple data components, such as performing correlation analysis on sales data components from two different regions. Sub-computation logic refers to partial computations that can be completed using only the data within a single data component, such as calculating the total sales revenue within each sales data component. Aggregation logic refers to computational steps used to combine the intermediate results generated by various sub-computation logics to arrive at the final result, such as adding the total sales revenue from two regions.

[0085] Specifically, this step is triggered when the data processing system determines multiple target data items based on the data request in step S203, and the request includes logic for jointly processing these data items. The query optimizer or computation task planning module in the data processing system analyzes the dependencies of this joint computation logic. It identifies which operations (such as filtering, internal aggregation) can be performed independently on each data item first, defining these operations as sub-computation logic. The remaining operations that require all sub-computation results to be available before they can be performed (such as Join, final SUM, or AVERAGE) are defined as aggregation logic.

[0086] In some embodiments, the decomposition of computational logic can be achieved in several ways: Optionally, rule-based decomposition, where the system pre-defines a set of decomposition rules for standard SQL queries, such as decomposing SELECT SUM(a.sales) FROM table_aa JOIN table_bb ON a.id=b.id into preprocessing logic on a and b respectively, and the final Join and SUM aggregation logic; Optionally, based on execution graph analysis, the entire computation task is represented as a directed acyclic graph (DAG), where subgraphs without cross-database dependencies constitute sub-computation logic, and the remaining parts constitute aggregation logic. It is understood that other methods can also be used to achieve logical decomposition, with the goal of maximizing the proportion of local computation; this is not limited here.

[0087] In some embodiments, certain computational logic may not be clearly decomposed into independent sub-logic, such as tasks requiring sorting of the entire dataset or calculating the median. In such cases, the data processing system identifies this type of logic as globally dependent logic. During decomposition, the system does not forcibly split it but instead marks it as an aggregate logic. Simultaneously, the system checks whether the amount of data required for subsequent global computation can be reduced by performing some filtering or sampling sub-computations first. If so, such sub-computation logic is generated; otherwise, the entire joint computation logic is treated as a single, indivisible aggregate task and processed centrally in subsequent steps.

[0088] In some embodiments, the data processing system dynamically schedules computational tasks. Specifically, the data processing system queries the metadata of individual data components to obtain an estimated data size and the available computing resources of the target storage node where each data component is located. Based on the complexity of the sub-computation logic, the estimated data size, and the available computing resources, the system calculates the estimated execution cost of each sub-computation logic. When the estimated execution cost exceeds a preset cost threshold, the corresponding sub-computation logic is identified as a complex computation logic. The system then pulls the data entity fragments required to execute the complex computation logic from the corresponding target storage node to a designated computing node to execute the complex computation logic on the designated computing node.

[0089] Among these, data size estimation refers to the estimate of the amount of data contained in a data file, such as the number of rows or bytes. Available computing resources refer to the amount of CPU, memory, and other resources currently idle on the target storage node. The complexity of sub-computation logic is a quantitative assessment of the time or space complexity of the algorithm. Estimated execution cost is a comprehensive indicator that combines data volume, algorithm complexity, and resource status to predict the execution time or resource consumption of a task. Complex computational logic refers to computational tasks whose estimated execution cost is too high and are unsuitable for execution on resource-constrained storage nodes. Designated computing nodes are resource-rich servers specifically designed for performing large-scale or complex computations.

[0090] Specifically, after decomposing the joint computation logic in step S205 and before generating computation instructions in step S206, the data processing system performs a cost assessment and scheduling process. For each sub-computation logic and its corresponding target data component, the system first queries the data component's metadata to obtain its size (e.g., 100 million rows) and queries the target storage node for its currently available resources (e.g., 2 idle CPU cores, 4GB available memory). The system also analyzes the type of the sub-computation logic (e.g., simple filtering is O(N), complex association may be O(N^2)). Based on these inputs, a cost model function calculates the estimated execution cost. For example, cost = f(complexity, data size) / g(available resources). Then, the system compares this cost with a preset cost threshold. If it exceeds the threshold, the sub-computation logic is marked as complex computation logic, and the system decides not to push it down, but instead generates a data fetch task, fetching all the data entity fragments required to execute the logic from the target storage node to a specified, powerful computing node for execution.

[0091] It should be noted that the cost model function, cost = f(complexity, data size) / g(available computing resources), is only an exemplary formula. In a specific implementation, it can also be a calibrated multiple linear regression model or a more complex machine learning model. The complexity of the sub-computation logic is quantified into a numerical value. The system predefines an operator complexity mapping table. For example, row-level filtering is assigned a complexity of 1, projection is assigned 0.5, simple aggregation (such as SUM, COUNT) is assigned 2, while complex operations involving sorting or joining are assigned higher values ​​based on their algorithmic complexity (such as O(NlogN)), for example, 10. Therefore, the function f(complexity, data size) can be concretized as f = complexity quantification * estimated data size (such as number of rows). The function g(available computing resources) is a comprehensive score of node capabilities, specifically in the form g = w1 * number of idle CPU cores + w2 * available memory size (GB), where w1 and w2 are weighting coefficients determined based on empirical data. Therefore, a more specific cost estimation formula is: Estimated execution cost = (Operator complexity quantification value * Number of data rows) / (w1 * Number of idle CPU cores + w2 * Available memory size). This cost value represents the estimated execution time or resource consumption unit. The system determines whether to mark the sub-computation logic as "complex computation logic" by comparing this calculation result with a dynamically adjusted threshold (e.g., based on the system's average task execution time).

[0092] In addition, in some embodiments, the data processing system optimizes complex computations in steps. That is, the data processing system decomposes complex computation logic into local preprocessing logic and centralized aggregation logic. The local preprocessing logic includes filtering or transformation operation logic to reduce the data size. The computation instructions containing the local preprocessing logic are sent to the corresponding target storage node to generate lightweight intermediate data with a data size smaller than the corresponding data entity fragment on the target storage node. The lightweight intermediate data is pulled from the target storage node to a designated computing node to execute the centralized aggregation logic on the designated computing node to obtain the final execution result.

[0093] Local preprocessing logic is the part of complex computational logic that can be separated, executed at the data source, and effectively reduces the amount of data. Centralized aggregation logic is the remaining part of complex computational logic that must be processed on a centralized node. Lightweight intermediate data is a dataset generated after local preprocessing, with a size much smaller than the original data entity fragments.

[0094] Specifically, when a sub-computation logic is identified as complex in the previous stage, the system does not immediately retrieve all the raw data. Instead, it first attempts to decompose the complex logic itself. The system analyzes the logic to see if a preprocessing step with data reduction effects can be extracted. For example, a complex analysis task might include an early filtering step (e.g., WHERE year=2025) or a projection step (e.g., selecting only 3 columns relevant to the analysis). The system defines this filtering or projection operation as local preprocessing logic and sends it as a computation instruction to the target storage node. The storage node executes this preprocessing logic, generating lightweight intermediate data containing only 2025 data and 3 columns. Then, the system retrieves only this lightweight intermediate data from the target storage node to the designated computation node. Finally, the remaining part of the original complex logic (i.e., centralized aggregation logic) is executed on the computation node to obtain the final result.

[0095] In some embodiments, the secondary decomposition of complex logic can be achieved in several ways: Optionally, based on query rewriting technology, the algebraic representation of the computational logic is analyzed, and operators that can reduce data volume, such as selection and projection, are moved forward as much as possible as local preprocessing logic; Optionally, for some specific statistical algorithms, their approximate calculation or sketch algorithm is used as preprocessing logic, for example, the HyperLogLog algorithm is used to calculate the approximate value of the cardinality locally, and only a very small amount of sketch data is transmitted to the central node for merging. It is understood that other methods can also be used to design preprocessing logic, the goal of which is to maximize data compression at the data source without affecting the final result or the accuracy of the result, which is not limited here.

[0096] In some embodiments, there may be situations where effective local preprocessing logic cannot be found, meaning the entire complex computational logic is an indivisible whole, and no local computation can reduce the data volume (e.g., cryptographic hashing of the entire data). The data processing system will identify this situation during the secondary decomposition stage. When analysis reveals that the logic does not contain any reduction operations such as selection, projection, or early aggregation, the decomposition process will fail. At this point, the system will revert to the strategy of the previous stage, that is, instead of attempting to generate preprocessing instructions, it will directly generate a data fetch task to fetch complete, unprocessed data entity fragments from the target storage node to the specified computing node for execution.

[0097] S206. Match each target data element with a calculation instruction containing the corresponding sub-calculation logic and calculation parameters.

[0098] The computation instruction is an executable command package sent to the target storage node, informing the node what operation to perform on the managed data. The computation parameters are the specific values ​​or conditions required to execute the sub-computation logic, such as a threshold in a filtering operation or a grouping key in an aggregation operation.

[0099] Specifically, after decomposing the sub-computation logic in step S205, the data processing system generates a corresponding computation instruction for each target data element. This instruction typically contains three parts: a unique semantic identifier for the target data element, the sub-computation logic to be executed (which can be a code snippet, function name, or predefined operator ID), and the computation parameters required to execute that logic. For example, for a target data element, its corresponding computation instruction might be: {target_id: uuid-123, logic: filter_and_sum, params: {filter_col: region, filter_val: East, sum_col: sales}}.

[0100] In some embodiments, the matching and generation of computation instructions can be achieved in several ways: Optionally, a portable script code (such as SQL or JavaScript) can be generated for each sub-computation logic and included as a string in the computation instruction; alternatively, a set of operator IDs can be predefined between the system and storage nodes, and the computation instruction only contains the operator ID and parameters. The storage node then calls the existing local operator based on the ID, which is more efficient and secure. It is understood that other methods can also be used to express and transmit computation instructions, and these are not limited here.

[0101] In some embodiments, there may be situations where a sub-computation logic depends on the intermediate results of another sub-computation logic (i.e., there is a sequence dependency). To address this, the data processing system constructs a dependency graph of sub-computation tasks (different from a data fragment dependency graph) when decomposing the logic. When generating computation instructions, the system does not issue all instructions at once, but rather issues them in batches according to the topological order of the dependency graph. Only after all the preceding dependent tasks of a task have been completed and returned results will the system send the computation instructions for that task (which may include the intermediate results of the preceding tasks as parameters) to the corresponding target storage node.

[0102] S207. Send the calculation instructions to the target storage nodes that host each target data component, and trigger the standardized operation interface of the corresponding target data component to execute the corresponding sub-computation logic on the target storage node to generate intermediate calculation results.

[0103] In this context, triggering the standardized operation interface of the corresponding target data component means that the execution of computation instructions is not carried out through an independent computation engine, but rather through the operation interface encapsulated within the data component itself. Intermediate computation results are temporary data, typically smaller in size than the original data, generated after the sub-computation logic is executed on a single data component.

[0104] Specifically, the data processing system distributes the computation instructions generated in step S206 to the target storage nodes hosting the corresponding target data components via the network. Upon receiving the computation instructions, the daemon process on the target storage node parses the instruction content. It does not directly manipulate the original data file, but instead calls the standardized operation interface of the target data component specified in the instruction, passing the sub-computation logic and parameters from the instruction as call parameters. The data component's interface implementation is responsible for loading the required data entity fragments, performing the computation, and returning an intermediate computation result. For example, the storage node calls the data component's `execute(logic, params)` method, which internally performs filtering and summation and returns a numerical value.

[0105] In some embodiments, computation pushdown can be implemented in several ways: Optionally, the standardized operation interface provides a general query or execute method that can interpret and execute a DSL (Domain-Specific Language) or script, with sub-computation logic passed in as a DSL; alternatively, the standardized operation interface pre-defines various high-performance atomic computation methods (such as filter, sum, groupBy), and the sub-computation logic is decomposed into consecutive calls to these atomic methods. It is understood that other methods can also be used to trigger and execute local computation, the key being to leverage the self-executing capability of data components; this is not limited here.

[0106] In some embodiments, a situation may arise where the target storage node's computing resources are exhausted, preventing it from completing sub-computation logic. To address this, the target storage node internally employs a resource manager and a task queue. Upon receiving a computation instruction, the resource manager assesses the estimated resource consumption of the task. If current resources are insufficient, the task enters the waiting queue. The node schedules tasks in the queue according to priority (e.g., FIFO or based on estimated task duration). Simultaneously, the node reports its high load status to the central data processing system, which may then avoid assigning tasks to that node in subsequent task allocations.

[0107] S208. Collect all intermediate calculation results from each target storage node, execute aggregation logic on its own computing node or a preset computing node, obtain the final calculation result, and send it to the requester.

[0108] In this context, "collection" refers to the data processing system receiving intermediate computation results from all target storage nodes that performed sub-computations. "Self-computing node" or "pre-defined computing node" refers to the server that performs the final aggregation operation; this could be the data processing system that initiated the task or a node specifically designed for aggregation computation. The final computation result is the final output of the entire collaborative computing task.

[0109] Specifically, after all target storage nodes complete their sub-computations, they send the generated intermediate computation results back to the aggregation node of the data processing system. The aggregation node waits until it receives all the expected intermediate results, or until some nodes time out and continue according to a preset strategy. Once the results are collected, the aggregation node initiates the aggregation logic defined in step S205. For example, if the aggregation logic is summation, it adds all intermediate results (the sums of each partition). If the aggregation logic is join, it performs a join operation on the two sets of received intermediate data. After the aggregation logic is completed, the final computation result is formatted and sent to the requester that initially initiated the data request.

[0110] In some embodiments, result collection and aggregation can be implemented in several ways: optionally, a pull mode can be used, where the aggregation node polls the status of each target storage node and pulls the result when it is completed; optionally, a push mode can be used, where the target storage node pushes the intermediate results to the specified aggregation node address after completing the computation. The execution of the aggregation logic can utilize in-memory computing frameworks (such as Spark and Flink) to handle large-scale intermediate results. It is understood that other methods can also be used to complete the aggregation phase, which are not limited here.

[0111] In some embodiments, a target storage node may fail to compute or become unreachable, preventing the aggregation node from collecting the intermediate results it should return. To address this, the data processing system implements a fault tolerance and retry mechanism. The aggregation node sets a timeout for each sub-computation task. If a node does not return a result within the timeout period, the aggregation node reports a failure to the central scheduler. The scheduler then attempts to reschedule the failed sub-computation task to another replica storage node of the same data file. If multiple retries fail, the system can choose to terminate the entire task based on business needs, or continue aggregation even with missing intermediate results (e.g., ignoring missing parts when calculating the sum, but marking the result as incomplete).

[0112] S209. Based on the data structure logic, the target data component is split into a logical control segment containing a standardized operation interface and one or more data entity segments.

[0113] Refer to step S105, which will not be repeated here.

[0114] S210. Transmit the logic control segment to the requester, so that the requester can start the standardized operation interface before receiving the data entity segment.

[0115] Refer to step S106, which will not be repeated here.

[0116] In some embodiments, the data processing system implements asynchronous non-blocking access. Specifically, the system sends an instantiation instruction to the requester, causing the requester to generate an interface proxy object locally based on the interface definition in the logical control segment. When the requester's application calls any method in the interface proxy object, the proxy object performs a check to determine if the required entity fragment already exists in the local cache. If the check is yes, the system forwards the requester's call request and parameters to the loaded standardized operation interface and returns the execution result of the standardized operation interface. If the check is no, the system sends an asynchronous response instruction to the requester, causing the requester to generate a preset result object and return it to the application. The system then sends a data retrieval request containing the required entity fragment to the target storage node, receives the returned data entity content, and adjusts the preset result object with the data entity content.

[0117] The interface proxy object is an object that runs locally on the requester and has the same method signature as the standardized operation interface. It acts as an intermediary layer between the application and the actual data operations. The local cache is a block of memory or disk space on the requester's device used to store downloaded data fragments. The asynchronous response instruction is a signaling message that notifies the caller that its request cannot be completed. The preset result object is a placeholder object that provides a temporary, potentially empty or default value return when data has not arrived, and this object can be updated with real data in the future.

[0118] Specifically, after the requesting party receives the logic control fragment, it dynamically generates an interface proxy object based on the interface definition (such as a JSONSchema or IDL file). Application code calls the proxy object's methods as if calling a local object. When a method is called, the proxy object first checks if the entity fragment required to execute the method already exists in the local cache. If it exists, it directly calls the loaded standardized operation interface to process the cached data and synchronously returns the result. If it does not exist, the proxy object does not block and wait, but instead returns a pre-defined result object to the application (such as a Promise object or an empty but structurally complete object), while simultaneously initiating a background fetch request for the required entity fragment from the target storage node. Once the background data fetch is complete and the data entity content arrives, the proxy object populates or updates the previously returned pre-defined result object with real data. This update operation may trigger registered callback functions in the application or update the UI.

[0119] In some embodiments, the asynchronous proxy mechanism can be implemented in several ways: Optionally, in a JavaScript environment, the Promise / async / await syntax can be used, where the proxy method returns a Promise that resolves upon arrival of data; alternatively, in object-oriented languages ​​(such as Java or C#), the proxy method returns a Future or Task object and provides a callback registration mechanism, allowing the application to register a callback function that is invoked upon task completion. It is understood that other programming paradigms can also be used to implement this asynchronous non-blocking mode, the core of which is to separate the invocation and completion points; this is not limited here.

[0120] In some embodiments, applications may need to know the data loading status (such as progress or whether an error has occurred). To address this, the pre-defined result object returned by the interface proxy object is designed to be more comprehensive. This object is not merely a placeholder for the result; it also exposes properties describing the loading status, such as `isLoading`, `error`, and `progress`. The proxy object continuously updates these status properties as it fetches data in the background. This allows the application's UI to be bound to these properties; for example, displaying a loading animation when `isLoading` is true, a progress bar when `progress` is updated, and an error message when `error` is not empty, providing users with more granular interactive feedback.

[0121] S211. During the operation of the standardized operation interface, the corresponding data entity fragments are retrieved from the target storage node and transmitted synchronously according to the external access command.

[0122] Refer to step S107, which will not be repeated here.

[0123] In some embodiments, the logical control segment includes a standardized operation interface and a data segment dependency graph describing the relationships between data entity segments; the data processing system performs intelligent prefetching, that is, when the standardized operation interface receives an external access instruction pointing to the first entity segment, the data processing system queries the data segment dependency graph in the logical control segment to obtain the second entity segment that has a dependency relationship with the first entity segment; generates a transmission task list containing the first entity segment and the second entity segment, and sets the transmission priority of the second entity segment to be higher than that of other unrequested entity segments; according to the transmission task list, initiates a data retrieval request for the first entity segment and the second entity segment to the target storage node.

[0124] The data segment dependency graph is a data structure stored within logical control segments, typically a graph or an association list. It records the logical dependencies between data entity segments; for example, in video data, a P-frame depends on its preceding I-frame or P-frame. The first entity segment is the data block currently being directly requested by the user. The second entity segment is the data block found in the dependency graph that is associated with the first entity segment. The transmission task list is a temporary, pending queue of data fetch requests, where each task contains the identifier of the entity segment to be fetched and its transmission priority.

[0125] Specifically, this process occurs on the requesting side. When an application requests data through a standardized operation interface (e.g., a video player drags to a specific point in time and requests the corresponding video data segment, i.e., the first entity segment), the interface's local implementation or proxy object first checks if a data segment dependency graph exists in the logical control segment. If it exists, it uses the identifier of the first entity segment as the key to search the graph for all entity segments (i.e., second entity segments) that have a dependency relationship with it. For example, when requesting a P-frame, the dependency graph will indicate that it depends on the previous keyframe (I-frame). Next, the system creates a transmission task list, setting the transmission priority of the first entity segment to the highest and setting the priority of all found second entity segments to the second highest, higher than all other unrequested segments. Finally, based on this priority-based task list, the system initiates a data retrieval request to the target storage node, which will prioritize sending the higher-priority segments.

[0126] In some embodiments, dependency prefetching can be implemented in several ways: Optionally, a batch request approach can be used, where all entity fragment IDs in the transmission task list are packaged and sent to the target storage node at once, and the node arranges the sending order according to priority; alternatively, a concurrent request stream approach can be used, where independent HTTP / 2 request streams are established for the first entity fragment and each second entity fragment, and the server's sending strategy is guided by setting stream priority parameters. It is understood that other methods can also be used to implement the prefetching mechanism, such as dynamically adjusting the number of prefetched fragments based on network conditions; this is not limited here.

[0127] In some embodiments, prefetched data segments may ultimately go unused, resulting in wasted bandwidth. To address this, the data processing system implements a prefetching strategy with feedback and learning capabilities. The requester's interface proxy object monitors whether the prefetched second entity segment is actually accessed within a short period. The system calculates the hit rate for each dependency. For dependencies with low hit rates, the system dynamically reduces their weight in future prefetching decisions, or, when network bandwidth is limited, completely avoids prefetching low-hit-rate dependency segments. Through this adaptive adjustment, the system can achieve a balance between improving access consistency and saving bandwidth.

[0128] It's important to note that the adaptive adjustment strategy here is based on a sliding window hit rate statistical model with a decay factor. Specifically, for each dependency edge in the data fragment dependency graph (e.g., from fragment A to fragment B), the system maintains two values ​​locally on the requester: a prefetch weight W_AB (initially 1.0) and a recent access timestamp T_AB. When fragment B is prefetched due to a request for fragment A, if fragment B is actually accessed within a set time window (e.g., 5 seconds), it's considered a hit, and the system boosts the weight using the update formula W_AB = W_AB + α * (1 - W_AB), where α is the learning rate (e.g., 0.1), which brings the weight closer to 1.0. If fragment B remains unaccessed after the time window, it's considered a miss, and the system penalizes the weight using the update formula W_AB = W_AB * β, where β is the decay factor (e.g., 0.95), causing the weight to decrease exponentially over time or with the number of misses. When initiating a prefetch decision, the system not only checks whether the dependency exists, but also determines whether its weight W_AB is higher than a dynamic threshold T_prefetch. This threshold itself is adaptive, adjusting based on the current network conditions (such as bandwidth and latency): when the network is good, the threshold is lowered to encourage more prefetching; when the network is congested, the threshold is raised to conserve bandwidth. In this way, the system can dynamically learn access patterns and make intelligent trade-offs between the immediacy of data access and the consumption of network resources based on the real-time network environment.

[0129] In some embodiments, the data processing system performs data-derived transmission optimization. Specifically, the data processing system reads the metadata of the target data file, extracts the source data hash value on which the metadata depends, and performs preprocessing and vector generation processing algorithm logic and parameters; sends a comparison instruction containing the source data hash value to the requester; when the system determines that the requester has basic data with the same source data hash value based on the matching signal returned by the requester, it prohibits the transmission of data entity fragments in the target data file; generates a reconstruction instruction package containing standardized operation interfaces, algorithm logic, and algorithm parameters, and sends the reconstruction instruction package to the requester, enabling the requester to generate the corresponding data entity fragment locally based on the basic data and the reconstruction instruction package.

[0130] The source data hash value is a unique checksum of the original underlying data upon which the data item is generated. Algorithm logic and parameters refer to the processing steps and configurations involved in generating the current data item from the underlying data, such as the structure and weights of a machine learning model. The comparison command is sent to the requesting party to query whether underlying data with a specific hash value exists locally. The reconstruction command package is a lightweight set of commands that tells the requesting party how to reproduce the target data item using its local underlying data.

[0131] Specifically, this optimization primarily targets derived data items (e.g., a feature vector set data item generated from an original image set). When such a data item is requested, the data processing system first reads the hash value of the original image set it depends on, as well as the model information (algorithm logic and parameters) used to generate the vectors, from its metadata. Next, instead of sending the data directly, the system sends a comparison instruction to the requester, which includes the hash value of the source data. Upon receiving the instruction, the requester searches its local data cache or storage for underlying data (the original image set) with the same hash value. If found, it returns a positive match signal to the system. Upon receiving this signal, the system decides not to transmit the large feature vector data entity fragment. Instead, it generates a reconstruction instruction package containing a standardized operation interface for manipulating local data and the model information (algorithm logic and parameters) for generating the vectors. This instruction package is sent to the requester, who, after loading it, calculates and generates the required data entity fragment locally using its existing original image set and the received instruction.

[0132] In some embodiments, local refactoring can be implemented in several ways: Optionally, the refactoring instruction package includes a reference to local underlying data and an executable script. The requesting party runs the script, which reads the local data and executes the transformation logic. Optionally, the requesting party's standardized operation interface proxy object can interpret the refactoring instructions. When the upper-layer application requests data, the proxy object intercepts the request, calls the local computing engine (such as a Python environment and machine learning library) to execute the algorithm in the refactoring instructions, generates data in real time, and returns it. It is understood that other methods can also be used to implement local refactoring, the core of which is to utilize the principle of data + code = result, transmitting code rather than the result; this is not limited here.

[0133] In some embodiments, the computational cost of local reconstruction by the requesting party may be too high, even slower than directly downloading the data. To address this, the data processing system records not only the algorithm logic but also a reconstruction cost estimate in its metadata. Before deciding whether to enable local reconstruction, the system compares this reconstruction cost estimate with the estimated network transmission time. The system will only adopt the local reconstruction scheme when the estimated local reconstruction time is significantly lower than the network transmission time. Furthermore, the system can provide the requesting party with a choice, allowing it to decide whether to choose bandwidth-saving local reconstruction or CPU-saving direct download based on its current device load.

[0134] In this embodiment, by encapsulating data and standardized operation interfaces into executable, addressable data components and constructing a complete data processing method from semantic retrieval and on-demand streaming to distributed computing pushdown, this solution transforms data from static, passive files into dynamic, service units, achieving deep decoupling of data from applications and storage. This method effectively solves core pain points in traditional data processing, such as data silos, inconsistent formats, low transmission efficiency, and massive data migration caused by the separation of computation and storage. It then achieves seamless data flow, instant availability, and efficient collaborative processing across heterogeneous systems, significantly reducing the cost and complexity of data integration and application, and improving the efficiency of data value release as a key production factor.

[0135] The data processing system in the embodiments of this invention is described below from a hardware processing perspective. Please refer to [link / reference needed]. Figure 3 This is a schematic diagram of the physical device structure of a data processing system in an embodiment of this application.

[0136] It should be noted that, Figure 3 The structure of the data processing system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0137] like Figure 3 As shown, the data processing system includes a CPU 301, which can perform various appropriate actions and processes according to a program stored in ROM 302 or a program loaded from storage section 308 into RAM 303, such as executing the methods described in the above embodiments. RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O interface 305 is also connected to bus 304.

[0138] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including hard disks, etc.; and communication section 309 including network interface cards such as LAN (Local Area Network) cards, modems, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0139] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by CPU 301, it performs the various functions defined in the present invention.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0141] Specifically, the data processing system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the data processing method driven by the data device provided in the above embodiment.

[0142] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the data processing system described in the above embodiments; or it may exist independently and not assembled into the data processing system. The storage medium carries one or more computer programs that, when executed by a processor of the data processing system, cause the data processing system to implement the data-driven data processing method provided in the above embodiments.

[0143] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0144] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

Claims

1. A data processing method driven by data components, characterized in that, Applied to a data processing system, the method includes: Acquire source data, and encapsulate the source data and the corresponding standardized operation interface into an executable data unit based on a standardized operation protocol to generate a data component with a unique semantic identifier; The data components are mapped to data semantic coordinates in the domain dimension, time dimension, and geographic dimension, and a data index tree is constructed for the unique semantic identifier, data storage address, and data semantic coordinates. Receive a data request, convert the semantic information of the data request into target semantic coordinates, and search in the data index tree to determine the target semantic identifier and target data component; Identify multiple storage nodes for the target data file and establish a transmission connection with the target storage node with the lowest latency; Based on the data structure logic, the target data component is divided into a logical control segment containing the standardized operation interface and one or more data entity segments; The logical control fragment is transmitted to the requester, who parses the logical control fragment and loads the interface execution environment locally, so that the requester starts the standardized operation interface before receiving the data entity fragment. Send an instantiation instruction to the requester, causing the requester to generate an interface proxy object locally based on the interface definition in the logical control segment; When the requesting application calls any method in the interface proxy object, the interface proxy object performs a check operation to determine whether the required entity fragment already exists in the local cache; If the check result is yes, then the requester's call request and call parameters are forwarded to the loaded standardized operation interface, and the execution result of the standardized operation interface is returned; If the check result is negative, an asynchronous response instruction is sent to the requester, causing the requester to generate a preset result object and return it to the application. Send a data retrieval request containing the required entity fragment to the target storage node, receive the returned data entity content, and adjust the preset result object with the data entity content; During the operation of the standardized operation interface, the corresponding data entity fragments are retrieved from the target storage node and transmitted synchronously according to the external access command.

2. The method according to claim 1, characterized in that, The logical control segment includes the standardized operation interface and a data segment dependency graph describing the relationships between the data entity segments; after the step of pulling and synchronously transmitting the corresponding data entity segments from the target storage node according to external access instructions during the operation of the standardized operation interface, the method further includes: When the standardized operation interface receives an external access instruction pointing to the first entity fragment, it queries the data fragment dependency graph in the logical control fragment to obtain the second entity fragment that has a dependency relationship with the first entity fragment; Generate a transmission task list containing the first entity fragment and the second entity fragment, and set the transmission priority of the second entity fragment to be higher than that of other unrequested entity fragments; According to the transmission task list, a data retrieval request for the first entity fragment and the second entity fragment is initiated to the target storage node.

3. The method according to claim 1, characterized in that, The data request includes joint computation logic for multiple target data items; after the step of determining multiple storage nodes for the target data items and establishing a transmission connection with the target storage node with the lowest latency, the method further includes: The joint computation logic is decomposed into one or more sub-computation logics that can be executed independently on a single data element and an aggregation logic; Match each target data element with a calculation instruction containing the corresponding sub-calculation logic and calculation parameters; The calculation instructions are sent to the target storage nodes that host each target data component, and the standardized operation interface of the corresponding target data component is triggered. The corresponding sub-computation logic is executed on the target storage node to generate intermediate calculation results. Collect all intermediate calculation results from each target storage node, execute the aggregation logic on its own computing node or a preset computing node to obtain the final calculation result, and send it to the requester.

4. The method according to claim 3, characterized in that, Following the step of decomposing the joint computation logic into one or more sub-computation logics that can be executed independently on a single data element and an aggregation logic, the method further includes: Query the metadata of the individual data components to obtain an estimated data size and the available computing resources of the target storage node where each data component is located; Based on the complexity of the sub-computation logic, the estimated data size, and the available computing resources, calculate the estimated execution cost of each sub-computation logic; When the estimated execution cost exceeds a preset cost threshold, the corresponding sub-computation logic is determined to be complex computation logic; The data entity fragments required to execute the complex computation logic are pulled from the corresponding target storage node to the designated computing node, so that the complex computation logic can be executed on the designated computing node.

5. The method according to claim 4, characterized in that, After determining that the corresponding sub-computation logic is complex computation logic when the estimated execution cost exceeds a preset cost threshold, the method further includes: The complex computational logic is decomposed into local preprocessing logic and centralized aggregation logic; the local preprocessing logic includes filtering operation logic or transformation operation logic to reduce the data size. The computation instructions containing the local preprocessing logic are sent to the corresponding target storage node to generate lightweight intermediate data with a data volume smaller than the corresponding data entity fragment on the target storage node. The lightweight intermediate data is pulled from the target storage node to the designated computing node to execute the centralized aggregation logic on the designated computing node and obtain the final execution result.

6. The method according to claim 1, characterized in that, After determining the addresses of multiple storage nodes of the target data file and the network latency parameters of each node, and establishing a transmission connection with the node address with the lowest latency, the method further includes: Read the metadata of the target data file, extract the source data hash value on which the metadata depends, and perform the algorithm logic and algorithm parameters for preprocessing and vector generation. Send a comparison instruction containing the hash value of the source data to the requesting party; When it is determined from the matching signal returned by the requester that the requester has basic data with the same hash value as the source data, the transmission of data entity fragments in the target data file is prohibited; A reconstruction instruction package containing the standardized operation interface, the algorithm logic, and the algorithm parameters is generated, and the reconstruction instruction package is sent to the requester, so that the requester can generate the corresponding data entity fragment locally based on the basic data and the reconstruction instruction package.

7. A data processing system, characterized in that, The data processing system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the data processing system to perform the method as described in any one of claims 1-6.

8. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the data processing system, the data processing system performs the method as described in any one of claims 1-6.

9. A computer program product, characterized in that, When the computer program product is run on the data processing system, the data processing system performs the method as described in any one of claims 1-6.

Images