Data processing method and device and electronic equipment

Abstract

The invention discloses a data processing method and device and electronic equipment. The method comprises the following steps: acquiring structured data, and storing the structured data in a primary key table in a lake storage engine; a metadata association table is created in the lake storage engine, description information of the unstructured data is stored in the metadata association table, and the metadata association table is associated with the primary key table through the associated primary key value; and receiving a feature vector obtained by calling a preset model to process the unstructured data by the stream processing engine, and storing the feature vector in the metadata association table. According to the data processing method and device, the technical problems that in the related technology, structured data and unstructured data are stored in different systems, and due to splitting of physical storage, the data processing flow is complex, and the efficiency is low are solved.

Description

Technical Field

[0001] This application relates to the field of data storage, and more specifically, to a data processing method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, AI model training is making great strides towards multimodal data fusion. Multimodal data fusion aims to comprehensively utilize the advantages of structured data (such as relational tables and time-series metrics) and unstructured data (such as images, text, audio, and video) to improve model performance and application effectiveness. However, current AI model training faces systemic architectural bottlenecks when dealing with this complex scenario, especially when handling the collaborative computation of structured and unstructured data, exposing core deficiencies and severely restricting the efficiency and quality of multimodal AI model training.

[0003] Current AI model training, especially in scenarios involving unstructured data (such as images, text, audio, and video), faces significant challenges. Unstructured data is typically stored in object storage (such as OSS, S3) or distributed file systems, while structured data is stored in traditional lake storage systems or data warehouses. This fragmentation of physical storage leads to complex and inefficient data processing workflows.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] This application provides a data processing method, apparatus, and electronic device to at least solve the technical problem in the related art where structured and unstructured data are stored in different systems, resulting in complex and inefficient data processing due to the fragmentation of physical storage.

[0006] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: acquiring structured data and storing the structured data in a primary key table in a lake storage engine; creating a metadata association table in the lake storage engine and storing descriptive information of unstructured data in the metadata association table, wherein the metadata association table is associated with the primary key table through an association primary key value; receiving a feature vector obtained by a stream processing engine calling a preset model to process the unstructured data, and storing the feature vector in the metadata association table.

[0007] Optionally, the stream processing engine calls a preset model to process unstructured data, including: encapsulating the preset model into a user-defined function and registering the user-defined function in the stream processing engine; in the stream processing engine, loading the unstructured data stored in object storage and processing the unstructured data by calling the user-defined function.

[0008] Optionally, the stream processing engine calls a preset model to process the unstructured data, including: capturing the change log stream of the lake storage engine and obtaining the capture result, wherein the change log stream records the data changes in the lake storage engine; if the capture result indicates that the description information of the unstructured data in the metadata association table has been updated, reading the updated description information in the change log stream; reading the updated data of the unstructured data from the object storage based on the updated description information; and calling a user-defined function to process the updated data.

[0009] Optionally, the method further includes: obtaining the input flow intensity of the stream processing engine; and adjusting the batch processing parameters of the pipeline in the stream processing engine based on the input flow intensity.

[0010] Optionally, adjusting the batch processing parameters of the pipeline in the stream processing engine based on the input flow intensity includes: determining a first value for the batch processing parameter when the input flow intensity is less than a first threshold; determining a second value for the batch processing parameter when the input flow intensity is greater than or equal to the first threshold and less than or equal to a second threshold, wherein the second value is greater than the first value and the second threshold is greater than the first threshold; and determining a third value for the batch processing parameter when the input flow intensity is greater than the second threshold, wherein the third value is greater than the second value.

[0011] Optionally, the method further includes: in the event of a failure in the stream processing engine, determining the breakpoint of the change log stream using the time travel function in the lake storage engine; determining the target description information to be updated after the breakpoint, and reading the target update data of unstructured data from the object storage based on the target description information; and re-invoking the user-defined function to process the target update data.

[0012] Optionally, the method further includes: when the preset model is updated from the first version to the second version, obtaining the model version information in the metadata association table; determining the first feature vector generated by the preset model of the first version based on the model version information; and calling a user-defined function to process the unstructured data corresponding to the first feature vector using the preset model of the second version.

[0013] Optionally, the feature vector includes text vectors and image vectors. The method further includes: obtaining a predefined query vector, wherein the query vector includes query text and query image; determining the account ID from the metadata association table, and retrieving the target text vector and target image vector of the account ID using a vector search function to determine the first similarity between the target text vector and the query text, and determining the second similarity between the target image vector and the query image, wherein the account ID is the associated primary key value; determining the account balance in the primary key table based on the account ID in the metadata association table; identifying account IDs with account balances exceeding a preset value as target account IDs; determining the target first similarity and target second similarity corresponding to the target account IDs, and performing weighted fusion of the target first similarity and target second similarity to obtain a comprehensive score; and identifying the account IDs with the highest comprehensive scores among the top preset number as risk account IDs.

[0014] According to another aspect of the embodiments of this application, a data processing apparatus is also provided, comprising: an acquisition module, configured to acquire structured data and store the structured data in a primary key table in a lake storage engine; a creation module, configured to create a metadata association table in the lake storage engine and store descriptive information of unstructured data in the metadata association table, wherein the metadata association table is associated with the primary key table through an association primary key value; and a receiving module, configured to receive a feature vector obtained by a stream processing engine calling a preset model to process the unstructured data, and store the feature vector in the metadata association table.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory for storing program instructions; and a processor connected to the memory for executing program instructions to perform the following functions: acquiring structured data and storing the structured data in a primary key table in a lake storage engine; creating a metadata association table in the lake storage engine and storing descriptive information of unstructured data in the metadata association table, wherein the metadata association table is associated with the primary key table through an association primary key value; and receiving a feature vector obtained by a stream processing engine calling a preset model to process the unstructured data and storing the feature vector in the metadata association table.

[0016] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described data processing method by running the computer program.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described data processing method.

[0018] In this embodiment, structured data is acquired and stored in the primary key table of the lake storage engine; a metadata association table is created in the lake storage engine, and the description information of unstructured data is stored in the metadata association table, wherein the metadata association table is associated with the primary key table through the association primary key value; the feature vector obtained by the stream processing engine calling the preset model to process the unstructured data is received, and the feature vector is stored in the metadata association table, thereby eliminating the physical storage separation between structured and unstructured data, thus achieving the technical effect of improving data processing efficiency. This solves the technical problem in related technologies where structured and unstructured data are stored in different systems, and the separation of physical storage leads to complex data processing flow and low efficiency. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a data processing method according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application;

[0022] Figure 3 This is a system processing flow overview diagram of an AI model streaming collaborative training framework based on a unified lake storage architecture according to an embodiment of this application;

[0023] Figure 4 This is a structural diagram of a data processing apparatus according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0027] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0028] User-defined functions (UDFs) are specific functions in programming languages ​​that allow users to write according to their own needs. They extend the functional boundaries of built-in system functions, enabling users to implement custom processing for complex business logic, data transformation, or specific computational requirements.

[0029] SST (Sort String Table File): SST files are the actual data storage units in an LSM tree. They are arranged in key order, allowing data to be queried and updated efficiently.

[0030] Time Travel: A feature that allows users to query snapshots of data at any point in time.

[0031] HNSW (Hierarchical Navigable Small World): An efficient indexing structure for approximate nearest neighbor search (ANN), specifically designed for quickly finding similar items in high-dimensional vector data.

[0032] IVF-Flat (Inverted-index-based Flat Index): An inverted-index-based indexing method widely used for efficient approximate nearest neighbor search on large-scale datasets.

[0033] In related technologies, the fragmentation of physical storage leads to complex and inefficient data processing flows, manifested in the following ways: First, the metadata of unstructured data (such as storage paths and basic attributes) is scattered across its corresponding structured data (such as business tags and associated IDs) and derived feature vectors, making efficient correlation, querying, and unified management difficult. This hinders the smoothness of the end-to-end training pipeline. For example, to "retrieve the embedded vector of a user's avatar," the system needs to jump between storage systems multiple times: first, obtain the user's relevant information from the structured data storage system; then, based on the associated foreign key information, search for the corresponding unstructured data in object storage or a distributed file system; and finally, perform feature extraction and vector matching. Second, there is the issue of version control disorder. Structured data can be managed through transaction version control to ensure data consistency and traceability. However, unstructured data lacks a native version mechanism. When model training requires retracing data snapshots at specific points in time, it is difficult to maintain consistency between the structured data version and the unstructured file state. Furthermore, the reading, parsing, and feature extraction (such as generating embedding vectors) of unstructured data usually rely on offline batch processing jobs (such as Spark), which can take up to several hours and become a bottleneck in the process. Moreover, similarity retrieval for unstructured data (such as vector-based image search) is inefficient and lacks an indexing mechanism that is deeply integrated with the storage layer.

[0034] In the feature engineering stage, feature extraction from unstructured data generally relies on offline batch processing. The raw data needs to go through a "ingestion-storage-batch processing-feature entry" chain, with end-to-end latency often reaching hours or even longer, which cannot meet the requirements of real-time or near-real-time training. In addition, for feature vectors generated from unstructured data, traditional solutions require exporting the vector data to a dedicated vector database (such as Milvus or Faiss) to build an index. This process introduces additional data synchronization overhead, system consistency maintenance costs, and architectural complexity. Moreover, cross-system queries result in high retrieval latency, making it difficult to coordinate with query operations for structured data.

[0035] Lake-based storage systems lack native support for vector data and the ability to deeply integrate vector indexes with structured data storage. This necessitates reliance on external systems for similarity retrieval, increasing the complexity of data flow. Furthermore, the combination of streaming processing frameworks and feature engineering is limited to simple data transformation, making it difficult to dynamically integrate deep learning models for real-time inference on unstructured data and hindering the streaming generation and updating of feature vectors from raw data. These technical bottlenecks render traditional AI training pipelines inefficient and bloated when processing heterogeneous data, failing to meet the stringent requirements of real-time machine learning and online feature platforms, which demand latency in the minutes or even seconds.

[0036] To address the problems existing in related technologies, embodiments of this application provide a data processing method that can be run on... Figure 1 The computer terminal shown is explained below.

[0037] The data processing method embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0041] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0042] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0043] In the above operating environment, this application provides a data processing method embodiment. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0044] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0045] Step S202: Obtain structured data and store the structured data in the primary key table of the lake storage engine.

[0046] In step S202 above, structured data refers to data with a fixed format or structure, such as tabular data in a relational database, including but not limited to user attribute information, transaction records, and indicator data. A lake storage engine (or lake storage system), such as Apache Paimon, is a big data storage solution designed to uniformly manage and efficiently process large-scale heterogeneous data. In a lake storage engine, structured data is stored in a primary key table. The primary key table is designed based on the concept of a primary key, which is a field that uniquely identifies each row of data. It can be a single field or a combination of multiple fields. In scenarios involving a large number of point queries, the primary key table facilitates fast data retrieval and updates because the storage engine behind the primary key table (such as Paimon) uses a storage architecture similar to an LSM (Log-Structured Merge Tree) structure. This architecture supports high-speed write operations and efficient read optimizations. Especially when data is frequently updated, the LSM tree maintains good performance while providing TimeTravel (historical data version query) functionality.

[0047] Step S204: Create a metadata association table in the lake storage engine and store the description information of unstructured data in the metadata association table. The metadata association table is associated with the primary key table through the association primary key value.

[0048] In step S204 above, unstructured data (such as images, text, audio, and video) is stored in object storage (such as OSS or S3), and data persistence is ensured through a multi-replica mechanism. The metadata association table is mainly used to register and describe key information about unstructured data. It is created in a lake storage engine (such as Apache Paimon) and stores the following types of descriptive information: 1. Object storage access path: The entry point for each piece of unstructured data, usually the complete URI (Uniform Resource Identifier) ​​of a file or object in the object storage service (such as AWS S3 or OSS), facilitating subsequent access and download; 2. Association primary key value: A bridge between unstructured and structured data, which can be a unique identifier (such as a user ID or transaction ID) of a business entity related to the unstructured data. This field is used to quickly locate related structured data records in the primary key table, achieving logical unification between structured and unstructured data; 3. Feature Vector: Reserved for feature vectors generated after AI models process unstructured data in the future. These vectors are the basic data for model training and intelligent retrieval, and are initially NULL; 4. Extended Metadata: Includes version number, last update time, etc. This information is used for data version control and historical record tracking to ensure data consistency and traceability. Metadata-related tables enable logically unified management and efficient relational queries of heterogeneous data.

[0049] Fields in the metadata association table may include, for example, the following:

[0050] CREATE TABLE unstructured_meta (

[0051] id BIGINT PRIMARY KEY, -- Associated primary key value (associated with the structured data key)

[0052] os_uri STRING NOT NULL, -- Object storage access path (e.g., "s3: / / bucket / image.jpg")

[0053] feature_vector ARRAY <float>-- Reserved feature vector column (initially NULL)

[0054] version_id BIGINT,

[0055] last_update_time TIMESTAMP(6) -- Metadata update time

[0056] WITH (...);

[0057] The physical location field records the object access path using the RFC 3986 standard URI format and supports HTTPS signed access; the predefined feature vector column (or reserved feature vector column) is declared as a fixed-length floating-point array type (such as ARRAY). <float>The physical storage adopts the IEEE 754 single-precision floating-point format, and the memory is aligned to the 64-byte boundary to optimize the SIMD instruction set for acceleration; the spatiotemporal governance fields include version_id (a version identifier based on Hybrid Logical Clock) and last_update_time to support nanosecond-level precision tracking.

[0058] When new unstructured data is written to object storage, atomic transactions insert records into the unstructured_meta table, ensuring that the URI is bound to the primary key; Paimon's Time Travel capability supports historical metadata backtracking, meeting data version governance requirements.

[0059] In the above description, the associated primary key value not only enables logical binding between unstructured and structured data, but also ensures that whenever unstructured data changes (such as newly uploaded files or updated records) during the data ingestion and processing flow, a record is added or updated in the metadata association table. This record includes the URI of the unstructured data, the latest feature vector, and the corresponding associated primary key value. Through the associated primary key value, related structured data information can be quickly retrieved in the primary key table, enabling efficient cross-modal data association queries.

[0060] Furthermore, storing the metadata and feature vectors of unstructured data in the same metadata association table simplifies the data processing workflow. It eliminates the need to switch between multiple systems, reducing data transfer frequency, avoiding redundant data copying, and saving computing resources and bandwidth. Through fields such as version number and update time, the metadata association table supports time travel, allowing queries to show the state of data at any historical point in time. This ensures version consistency between structured and unstructured data during model training and data analysis, preventing erroneous analysis results due to data inconsistencies.

[0061] Step S206: Receive the feature vector obtained by the stream processing engine calling the preset model to process the unstructured data, and store the feature vector in the metadata association table.

[0062] In step S206 above, a dynamic model loading framework is implemented in the Flink SQL engine, including: loading TensorFlow / PyTorch models in isolation through Plugin ClassLoader; GPU memory pooling management to support elastic allocation of GPU memory for concurrent model inference.

[0063] The following is an example of the logic implementation of a user-defined function (UDF):

[0064] class ImageEmbeddingUDF extends ScalarFunction {

[0065] public float[] eval(String modelName, String ossUri) {

[0066] / / 1. Asynchronously read image byte stream from object storage

[0067] byte[] imageData = OssClient.fetch(ossUri);

[0068] / / 2. Load the specified model from the model repository

[0069] DLModel model = ModelRegistry.get(modelName);

[0070] / / 3. GPU Inference (Batch Processing Optimization)

[0071] return model.inference(imageData);

[0072] }

[0073] }

[0074] Leveraging an extended Flink engine and AI User-Defined Functions (AI UDFs), a streaming feature engineering pipeline is constructed. This pipeline consumes real-time data change log streams, dynamically loads pre-defined models (such as deep learning models) to perform inference on unstructured data, generates feature vectors, and writes them back to the Paimon table (i.e., the primary key table). Specifically, the stream processing engine (such as Apache Flink) subscribes to and consumes real-time data streams from unstructured data storage systems (such as object storage). Whenever new unstructured data (such as images or audio files) is uploaded, the stream processing engine immediately retrieves the updated information for this data, such as the file's storage path and related metadata. After retrieving the unstructured data, the stream processing engine dynamically invokes pre-deployed AI models (i.e., the aforementioned pre-defined models). These pre-defined models are used to extract feature vectors from the unstructured data; for example, an image recognition model can convert an input image into a vector form describing its content and features. The selection and invocation of models are implemented within the stream processing engine through AI UDFs, allowing the application of deep learning inference capabilities to the data stream in real time. After processing the unstructured data, the AI ​​model generates feature vectors. The stream processing engine writes the generated feature vectors along with metadata about the unstructured data (such as object storage paths, associated primary key IDs, etc.) back to the metadata association table in the lake storage engine.

[0075] The above process ensures that the feature vectors are updated in real time as the unstructured data is updated, avoiding the problem of feature vector lag in traditional batch processing, and improving the freshness of feature data and the timeliness of model training.

[0076] Through steps S202 to S206 above, this embodiment of the application uses Apache Paimon as a unified storage platform. By deeply integrating the strong metadata association mechanism between the structured data primary key table and the unstructured data object storage, combined with the dynamic feature generation capability of the streaming computing engine Flink and Paimon vector indexing technology, it achieves minute-level latency optimization from data ingestion to model training. This eliminates the physical storage separation between structured and unstructured data, thereby improving data processing efficiency. Furthermore, it solves the technical problem in related technologies where structured and unstructured data are stored in different systems, leading to complex and inefficient data processing due to physical storage separation. The following is a further explanation.

[0077] In step S206 of the above data processing method, the stream processing engine calls a preset model to process unstructured data, including: encapsulating the preset model into a user-defined function and registering the user-defined function in the stream processing engine; in the stream processing engine, loading the unstructured data stored in object storage and processing the unstructured data by calling the user-defined function.

[0078] In some embodiments of this application, a preset model is obtained and encapsulated into a user-defined function (UDF) that can be called by the stream processing engine. For example, in Flink, the UDF can be written in Java or Python, and the methods of calling the model in the deep learning framework can be implemented inside the function. The encapsulated UDF needs to be registered with the stream processing engine to make it a usable operation function. When unstructured data enters the stream processing framework, the framework can directly call the registered UDF without additional network requests or service calls to obtain the model inference results, thereby greatly improving processing speed and efficiency. The stream processing engine subscribes to real-time data streams from object storage, which can be newly uploaded files or updates to existing data. Data usually enters in the form of events, each carrying reference information of unstructured data (such as storage path) and possible metadata. For each received event, the stream processing engine loads the unstructured data based on the provided object storage path. After the unstructured data is loaded, the stream processing engine calls the previously registered UDF function, passing the loaded data as input to the model to perform feature extraction or transformation tasks. The UDF is responsible for decoding the data, calling the model for inference, and generating output results, such as feature vectors. After processing the data, UDF generates feature vectors or other structured outputs. The output results are written back to the metadata association table in the lake storage engine for subsequent data processing.

[0079] In step S206 of the above data processing method, the stream processing engine calls a preset model to process the unstructured data, including: capturing the change log stream of the lake storage engine and obtaining the capture result, wherein the change log stream records the data changes in the lake storage engine; when the capture result indicates that the description information of the unstructured data in the metadata association table has been updated, reading the updated description information in the change log stream; reading the updated data of the unstructured data from the object storage according to the updated description information; and calling a user-defined function to process the updated data.

[0080] In some embodiments of this application, whenever data in the lake storage engine changes (added, updated, or deleted), the system automatically generates a ChangeLog stream (i.e., the aforementioned change log stream). The ChangeLog stream records the specific details of the data changes, including the change type, timestamp, primary key value, and modified fields. To respond to data changes in real time, the stream processing engine subscribes to the ChangeLog stream generated by the lake storage engine. The subscription mechanism ensures that the stream processing engine can promptly capture any data changes, thereby providing the necessary information input for real-time data processing. After receiving the ChangeLog stream, the stream processing engine parses the events in the stream to determine if any unstructured data description information in the metadata association table has been updated. If it is confirmed that the description information in the metadata association table has been updated, the stream processing engine will directly read the corresponding unstructured data update portion from the object storage system based on the object storage path or other location information contained in the updated description information. After reading the updated portion of the unstructured data, the stream processing engine will call the previously encapsulated and registered user-defined functions (UDFs) to process this updated data. When the pre-defined model in UDF receives updates to unstructured data through the UDF interface, it immediately generates features, such as embedding vectors for newly uploaded images and word vectors for the latest text segment. After processing, the stream processing engine writes the generated feature vectors along with the descriptive information of the unstructured data back to the metadata association table in the lake storage engine, completing the feature data update.

[0081] By subscribing to and parsing the ChangeLog stream, the stream processing engine can respond instantly to data changes, and the generation of feature vectors for unstructured data is also performed in real time, avoiding the lag of feature data and improving the real-time performance and accuracy of data processing.

[0082] The above data processing method further includes: obtaining the input flow intensity of the stream processing engine; and adjusting the batch processing parameters of the pipeline in the stream processing engine based on the input flow intensity.

[0083] In some embodiments of this application, the stream processing engine continuously monitors the input traffic intensity of the data stream it processes, i.e., the amount of data received per unit time. Through statistical analysis, the stream processing engine can determine the intensity of the current input traffic, for example, distinguishing between low traffic (light load), medium traffic (normal), and high traffic (heavy load). Batch processing parameters are adjusted according to different input traffic intensities.

[0084] In the above steps, adjusting the batch processing parameters of the pipeline in the stream processing engine based on the input flow intensity includes: determining the batch processing parameter as a first value when the input flow intensity is less than a first threshold; determining the batch processing parameter as a second value when the input flow intensity is greater than or equal to the first threshold and less than or equal to a second threshold, wherein the second value is greater than the first value and the second threshold is greater than the first threshold; and determining the batch processing parameter as a third value when the input flow intensity is greater than the second threshold, wherein the third value is greater than the second value.

[0085] In some embodiments of this application, the stream processing engine dynamically adjusts its batch processing parameters, primarily batch size and triggering conditions, based on the intensity of the input traffic. This strategy allows the system to automatically optimize processing efficiency under different load conditions, reduce latency, and avoid resource waste.

[0086] When the input traffic intensity is less than the first threshold (i.e., light load), a smaller batch processing parameter value (i.e., the first value) is determined. This means the stream processing engine will process data in groups of smaller amounts or adopt a strategy of processing single data items in real time, thereby minimizing latency and ensuring real-time data processing. When the input traffic intensity increases to greater than or equal to the first threshold but not exceeding the second threshold (i.e., medium load), the system will determine a larger batch processing parameter value than the first value (i.e., the second value). This can improve data processing throughput while remaining within an acceptable latency range, ensuring a balance between efficiency and real-time performance under medium traffic. If the input traffic intensity continues to increase and exceeds the second threshold (i.e., heavy load), the batch processing parameter value is further increased to a third value to fully improve processing efficiency and resource utilization. Under high traffic, a larger batch processing parameter can significantly reduce the unit cost of processing each batch of data. Through batch processing optimization, data processing stability and efficiency can be maintained even in high-concurrency scenarios.

[0087] The following examples illustrate this: In real-time risk control scenarios, when the system detects a low inflow rate of transaction events (e.g., less than 100 transactions / second), the batch processing parameter can be set to a small first value, such as 1. The trigger condition is that each data item is processed immediately upon arrival, ensuring that the latency is controlled within tens of milliseconds to meet the needs of real-time risk assessment. As the traffic intensity increases to a medium level (e.g., 100-1000 transactions / second), the batch processing parameter is dynamically adjusted to a second value (e.g., 16). Combined with a time window and data volume triggering mechanism (e.g., triggering processing every 100ms or when 16 data items are collected), both the real-time performance of data processing and the resource utilization rate are maintained. Under high traffic intensity (e.g., exceeding 1000 transactions / second), the batch processing parameter is further adjusted to a third value (e.g., 256). A data volume threshold-priority triggering strategy (e.g., starting processing when 256 data items are reached or a timeout of 300ms) is used to stabilize the latency (within 500ms), avoid system overload, and improve the utilization efficiency of GPU / CPU resources through batch processing. This strategy enables the streaming feature engineering engine to maintain optimal performance under different traffic conditions by dynamically switching processing modes. Compared with fixed batch processing solutions, resource utilization is improved by 25%-40%, and latency fluctuation is reduced by more than 60%.

[0088] This dynamic adjustment strategy ensures that the stream processing engine can self-adjust based on the intensity of real-time input traffic, thereby optimizing resource utilization and processing efficiency while maintaining real-time processing capabilities.

[0089] In the above data processing method, the method further includes: in the event of a failure in the stream processing engine, determining the breakpoint of the change log stream through the time travel function in the lake storage engine; determining the target description information to be updated after the breakpoint, and reading the target update data of unstructured data from the object storage based on the target description information; and re-invoking the user-defined function to process the target update data.

[0090] In some embodiments of this application, when the stream processing engine supports a fault recovery mechanism, based on Flink's Checkpoint mechanism and combined with Paimon's TimeTravel capability, the system automatically locates the breakpoint in the Changelog stream when the job restarts, and re-invokes the UDF to generate feature vectors for the unstructured data within the fault interval, ensuring exactly-once semantics. When the model version is upgraded, the system identifies the affected records through metadata fields, triggering an incremental recalculation process to avoid reprocessing all data.

[0091] Specifically, when the stream processing engine fails, the system immediately detects this state change. The stream processing engine uses its checkpoint mechanism to save the processing state before the failure. However, to accurately determine which data has been processed and which has not, the time travel function of the lake storage engine is needed. The time travel function allows querying the state of data at any point in time. Therefore, after the stream processing engine fails, this function can be called to determine the time point of the last successfully processed data, i.e., the breakpoint of the change log stream. For example, by querying the last_update_time field in the metadata association table, the timestamp of the last successful update before the failure can be found. After determining the breakpoint, all change description information from the breakpoint to the present is determined, i.e., the description information of the data that was not processed by the stream processing engine during the failure period. This yields the target description information, which may include the object storage path of unstructured data, the association key with structured data, and other metadata. After obtaining the target description information, the specific unstructured data update content, i.e., the target updated data, can be read from object storage based on this information. The reading process can rely on the object storage API to ensure that the required unstructured data to be updated is accurately obtained. The target updated data read from object storage is processed by re-invoking previously registered user-defined functions (UDFs). This process is similar to normal data processing, but specifically compensates for data affected during a failure to ensure data consistency and integrity. The processing results are written back to the lake storage's metadata association table, and for the generated feature vectors, incremental updates to the vector index are triggered to ensure the integrity of the data processing chain and the effectiveness of the vector retrieval function.

[0092] Specifically, the index is automatically updated through a vector writeback trigger, forming a data loop. An example is as follows:

[0093] CREATE PIPELINE realtime_feature AS

[0094] INSERT INTO risk_vectors

[0095] SELECT

[0096] user_id,

[0097] VIDEO_EMBEDDING( -- Video Feature Extraction UDF)

[0098] model_name='i3d-kinetics',

[0099] os_path=video_uri,

[0100] params='sample_rate=8' -- Number of frames sampled per second

[0101] AS action_vector,

[0102] AUDIO_TO_TEXT( -- Speech Recognition UDF)

[0103] model_name='whisper-large',

[0104] audio_uri=audio_url

[0105] AS transcript

[0106] FROM live_stream

[0107] WHERE event_type = 'VIDEO_CALL'

[0108] In the above data processing method, the method further includes: when the preset model is updated from the first version to the second version, obtaining the model version information in the metadata association table; determining the first feature vector generated by the preset model of the first version based on the model version information; and calling a user-defined function to process the unstructured data corresponding to the first feature vector using the preset model of the second version.

[0109] In some embodiments of this application, when a preset model is updated from one version to another, it is necessary to ensure that all relevant feature vectors are updated to reflect the latest model understanding and processing capabilities. The metadata association table stores not only descriptive information and feature vectors of unstructured data, but also model version information. When the preset model is updated from version one to version two, the system queries the metadata association table to find the feature vectors generated by the first version model. A user-defined function (UDF) is called, passing in the descriptive information of the unstructured data that needs feature vector updating. The UDF reads the unstructured data from object storage, processes it using the updated model, and generates new feature vectors. The new feature vectors are written back to the metadata association table, overwriting the original first feature vectors, and the model version information field is updated to reflect the model version used when the new feature vectors were generated.

[0110] The above process ensures the transparency of model updates to the data processing layer. The model version information in the metadata association table records the generation history of feature vectors, making it easy to identify feature vectors that need to be updated. Through an incremental recalculation strategy, the system only recalculates feature vectors for unstructured data affected by model version updates, rather than performing a full recalculation of the entire dataset, thus saving computing resources and storage space.

[0111] In a cross-border financial risk control scenario, suppose the text understanding model for transaction documents is upgraded from version 1.0 to version 2.0. The new version includes an improved relation extraction algorithm, enabling more accurate understanding of entities and relationships within the document. The system queries the metadata association table `transaction_docs` to find the `text_embedding` feature vector generated by the version 1.0 model. Then, the system calls the registered UDF function `cross_modal_embed`, passing in the new version model name (e.g., 'model-2.0'), and reads the transaction document data related to the `text_embedding` feature vector generated by the version 1.0 model from object storage, regenerating the feature vector. The newly generated feature vector is written back to the `transaction_docs` table, and the `model_version` field is updated to 2.0, ensuring that subsequent retrieval and training tasks can use the feature vector generated by the latest version model, improving the accuracy and response speed of the risk control model.

[0112] In the above data processing method, the feature vector includes text vectors and image vectors. The method further includes: obtaining a predefined query vector, wherein the query vector includes query text and query image; determining the account ID from the metadata association table, and retrieving the target text vector and target image vector of the account ID through a vector search function to determine the first similarity between the target text vector and the query text, and the second similarity between the target image vector and the query image, wherein the account ID is the associated primary key value; determining the account balance in the primary key table based on the account ID in the metadata association table; identifying account IDs with account balances exceeding a preset value as target account IDs; determining the target first similarity and target second similarity corresponding to the target account IDs, and performing weighted fusion of the target first similarity and target second similarity to obtain a comprehensive score; and identifying the account IDs with the highest comprehensive scores among the top preset number as risk account IDs.

[0113] In some embodiments of this application, the Paimon-SST storage format supports the construction and incremental updating of vector indexes (such as HNSW and IVF-Flat), providing high-performance similarity retrieval capabilities directly at the lake storage layer, thus avoiding dependence on external vector databases.

[0114] Specifically, a dedicated section for vector indexes is added to the Paimon SSTable file format, including: an index metadata header recording the index type (HNSW / IVF-Flat), vector dimension, and distance metric; a hierarchical data structure storing the multi-level graph structure of the HNSW index in a compact binary format; and an incremental marker bitmap identifying the location of newly added / deleted vectors, supporting minute-level incremental updates. In terms of storage layout, the raw vector data is physically adjacent to the primary key column, allowing a single I / O to load both business attributes and vector values ​​simultaneously, reducing retrieval I / O overhead by 70%. The index update process is automatically triggered by monitoring the Write-Ahead Log (WAL). Update strategies include: online appending of the index during small batch writes (<100ms latency); and background merging of the compaction thread to generate an optimized index during large batch updates, avoiding query blocking. Native vector retrieval operations are supported by extending SQL syntax, for example:

[0115] -- Find the 10 records most similar to the target vector

[0116] SELECT id, oss_uri, ANN_SCORE(feature_vector, [0.2,0.5,...]) ASsimilarity

[0117] FROM unstructured_metaWHERE VECTOR_SEARCH(feature_vector, [0.2,0.5,...], top_k=10)

[0118] In terms of performance assurance, deep index integration allows retrieval to skip unnecessary data copying, resulting in a performance improvement of more than 5 times compared to exporting to Faiss.

[0119] For example, when dealing with complex scenarios such as cross-border financial risk control, by combining structured data, such as account balances, with feature vectors (text vectors and image vectors) from unstructured data, the system can more comprehensively assess the risk level of an account. This will be explained below.

[0120] First, a predefined query vector is obtained, including query text and query image. This query vector is used to calculate similarity with feature vectors in the metadata association table. The query vector can be a vector representation of a known risk pattern or a vector description of a specific event. The account ID field in the metadata association table is used to locate accounts of interest. The account ID serves as the primary key value for associating structured and unstructured data. For each account ID, the target text vector and target image vector are retrieved. A vector search function (such as VECTOR_SEARCH) is used to calculate the first similarity between the target text vector and the query text, and the second similarity between the target image vector and the query image. The similarity calculation can be based on a specific distance metric (such as cosine similarity or Euclidean distance) to reflect the degree of similarity between feature vectors. Based on the account ID in the metadata association table, the account balance information in the primary key table is queried. Account IDs with balances exceeding a preset value are identified as preliminary risk account IDs, i.e., the aforementioned target account IDs. For the initially selected target account IDs, the first similarity (text vector similarity) and the second similarity (image vector similarity) are further fused, and a comprehensive score for each account is calculated by weighted summation. Based on the comprehensive score obtained by weighted fusion, a predetermined number of account IDs (e.g., the top 50) with the highest comprehensive scores are finally selected. These account IDs are the risk account IDs, representing the set of accounts with the highest risk under the comprehensive analysis of structured and unstructured data.

[0121] By combining text vectors and image vectors for similarity calculation, the system can comprehensively analyze structured and unstructured data, conduct more comprehensive risk assessments, identify potential fraud or abnormal behavior, and improve the accuracy and comprehensiveness of risk detection.

[0122] The data processing method provided in this application can be applied to a streaming collaborative training framework based on a unified lake storage architecture, and can achieve the following objectives: First, by constructing a new paradigm for data governance and extending Apache... The Paimon lake storage engine achieves unified management of structured data (primary key tables), unstructured data (object indexes), and derived feature vectors, completely eliminating data silos at the storage layer and reducing cross-modal relational query latency to the second level. Secondly, it breaks through the limits of real-time feature engineering by deeply integrating native AI UDFs into the Flink engine, supporting dynamic model loading and real-time feature vector generation in streaming pipelines, compressing unstructured data processing latency from hours to minutes, laying the data foundation for online training. Thirdly, it reconstructs the vector retrieval technology stack, designing the Paimon-SST format to natively support vector indexes (such as HNSW), making similarity retrieval an inherent capability of lake storage, eliminating cross-system data transfer, improving retrieval performance by more than 5 times, and reducing architectural complexity by 60%. Fourthly, it defines a new end-to-end training standard, connecting the entire chain of "data ingestion → feature streaming computation → near real-time index updates → sample stream supply," reducing the end-to-end latency from raw data to training-ready state to within 10 minutes, providing minute-level AI capabilities for scenarios such as real-time recommendation and risk monitoring.

[0123] To better understand the data processing method provided in the embodiments of this application, the following description is based on an embodiment of real-time generation and end-to-end training of multimodal features for cross-border financial risk control based on a unified lake storage architecture.

[0124] In cross-border financial risk control scenarios, financial institutions need to conduct real-time risk assessments on heterogeneous data containing basic account information (structured data) and transaction documents (contracts, invoices, and other unstructured data). Traditional solutions store account data in relational databases or data warehouses, and transaction documents in object storage. These are loosely linked through metadata, leading to reliance on offline batch processing for feature extraction (with latency of several hours) and on external vector databases for cross-modal retrieval (with complex architecture and high latency). This fails to meet the real-time identification and response requirements for high-risk accounts in cross-border transactions (such as detecting abnormal transactions in accounts with millions of yuan in balance). This practical application addresses this by constructing a unified lake storage architecture to achieve logically unified management of structured account data and unstructured transaction documents. It utilizes streaming feature engineering to generate text / image embedding vectors in real time, combines vector indexing technology to achieve efficient retrieval of multimodal risk features, and builds an end-to-end training pipeline based on real-time features, compressing the risk model update cycle to minutes and providing millisecond-level real-time risk control capabilities for cross-border financial businesses. The operational steps include:

[0125] 1. Data Modeling and Storage Initialization: A two-tier storage architecture is constructed to achieve unified management of structured information from cross-border accounts and unstructured data from transaction documents. First, a structured account primary key table, `global_accounts`, is created, using `account_id` as the primary key to store core information such as international bank account numbers, SWIFT codes, and real-time balances. Data distribution is optimized through partitioning with `swift_code` and bucketing with `account_id`, and a full-compaction mechanism is enabled to provide near real-time data change streams for downstream streaming processing. Second, an unstructured transaction document meta-table, `transaction_docs`, is created, using UUID as the primary key and associated object storage path, and predefined with a 1024-dimensional text vector column (`text_embedding`) and a 768-dimensional image vector column (`image_embedding`). This table adopts the Paimon-SST-vec storage format of this application, with a built-in HNSW vector index. It is strongly associated with the account primary key table through a foreign key (i.e., the associated primary key value) `account_id`, achieving cross-modal logical binding between account information and transaction documents. The specific implementation process is as follows:

[0126] -- Create the account master table (structured)

[0127] CREATE TABLE global_accounts (

[0128] account_id DECIMAL(20) PRIMARY KEY, -- International Bank Account

[0129] swift_code CHAR(11), -- SWIFT encoding

[0130] balance DECIMAL(24,2) -- Real-time balance

[0131] ) WITH (

[0132] 'partition' = 'swift_code', -- SWIFT code partition

[0133] 'bucket-key' = 'account_id', -- Bucketing by account ID

[0134] 'changelog-producer' = 'full-compaction' );

[0136] -- Create a transaction document metatable (unstructured)

[0137] CREATE TABLE transaction_docs (

[0138] doc_uuid CHAR(36) PRIMARY KEY, -- UUID identifier

[0139] account_idDECIMAL(20) REFERENCES global_accounts(account_id),

[0140] doc_category VARCHAR(32), -- Document type (contract / invoice)

[0141] os_location VARCHAR(1024), -- Object storage path

[0142] text_embedding ARRAY <float>(1024), -- Text vector (1024 dimensions)

[0143] image_embedding ARRAY <float>(768) -- Image vector (768 dimensions)

[0144] ) WITH (

[0145] 'file.format' = 'paimon-sst-vec', -- Vector-enhanced format

[0146] 'vector.index.type' = 'HNSW', -- Indexing algorithm

[0147] 'vector.ef_construction' = '300', -- Index construction parameter

[0148] 'vector.M' = '48' -- Number of inter-layer connections );

[0150] 2. Streaming Multimodal Feature Generation: A real-time feature generation pipeline is built based on Flink to perform cross-modal processing on newly uploaded transaction documents (such as contract PDFs and scanned invoices). A custom operator loads text and image data from object storage in parallel, and uses a thread pool to concurrently call a pre-trained model for GPU inference to generate feature vectors of the corresponding dimensions. This operator is encapsulated as a user-defined function (UDF) `cross_modal_embed`, registered in the stream processing engine, and then consumes the change log stream of the `transaction_docs` table in real time using declarative statements (processing only newly added data `op_type='INSERT'`). The generated text and image vectors (i.e., feature vectors) are written back to the reserved column of the metadata association table. This process, through parallel I / O and inference optimization, controls the feature generation latency of a single document to the minute level. The specific implementation process is as follows:

[0151] # Flink Python API defines cross-modal feature flow

[0152] class CrossModalFeature(StreamOperator):

[0153] def __init__(self):

[0154] self.text_model = ModelLoader("text-embedding-003")

[0155] self.image_model = ModelLoader("clip-vit-b32")

[0156] def process_element(self, row: Row, ctx: RuntimeContext):

[0157] # Parallel acquisition of raw text / image data

[0158] text_data = OssClient.get(row.oss_location + " / text.pdf")

[0159] image_data = OssClient.get(row.oss_location + " / scan.jpg")

[0160] # GPU Parallel Inference

[0161] with ThreadPoolExecutor(max_workers=2) as executor:

[0162] text_future = executor.submit(self.text_model.infer, text_data)

[0163] image_future = executor.submit(self.image_model.infer, image_data)

[0164] # Construct the output line

[0165] yield Row(

[0166] account_id=row.account_id,

[0167] text_vec=text_future.result(),

[0168] image_vec=image_future.result() )

[0170] # Register UDF and execute

[0171] t_env.create_temporary_function(

[0172] "cross_modal_embed",

[0173] udf(CrossModalFeature(),

[0174] result_type=DataTypes.ROW([

[0175] DataTypes.FIELD("account_id", DataTypes.DECIMAL(20)),

[0176] DataTypes.FIELD("text_vec", DataTypes.ARRAY(DataTypes.FLOAT())),

[0177] DataTypes.FIELD("image_vec", DataTypes.ARRAY(DataTypes.FLOAT())) ]) ) )

[0181] 3. Real-time Risk Retrieval System: In the risk assessment stage, multimodal joint retrieval is achieved by extending SQL syntax. First, a vector search is performed on the `transaction_docs` table, using the `VECTOR_SEARCH` function to simultaneously retrieve text and image vectors. The top 500 similarity results are used for initial screening. The search results are then JOINed with the `global_accounts` table to filter high-value accounts with balances exceeding a preset value. A weighted fusion strategy (e.g., text similarity weight 0.6, image similarity weight 0.4) is applied to these high-value accounts to calculate a comprehensive (risk) score. The top 50 records (i.e., the preset number) are output in descending order of the comprehensive (risk) score. This embodiment integrates the vector index deep into the Paimon storage layer, avoiding the overhead of exporting data to an external vector database in traditional solutions. This reduces the multimodal retrieval response time for millions of data points to sub-second levels, significantly improving the timeliness of risk identification. The specific implementation process is as follows:

[0182] -- Multimodal risk feature retrieval

[0183] SELECT a.swift_code, a.balance,

[0184] (0.6 * text_sim + 0.4 * image_sim) AS risk_score

[0185] FROM (

[0186] SELECT account_id,

[0187] ANN_SCORE(text_embedding, query_text) AS text_sim,

[0188] ANN_SCORE(image_embedding, query_image) AS image_sim

[0189] FROM transaction_docs

[0190] WHERE VECTOR_SEARCH(

[0191] text_embedding, query_text,

[0192] image_embedding, query_image, -- Multi-column combined search

[0193] fusion_strategy='WEIGHTED', -- Fusion strategy

[0194] top_k=500 )

[0196] ) features

[0197] JOIN global_accounts a ON features.account_id = a.account_id

[0198] WHERE a.balance> 1,000,000 -- Accounts with a balance in the millions

[0199] ORDER BY risk_score DESC

[0200] LIMIT 50;

[0201] 4. End-to-End Training Pipeline: A real-time training system based on PyTorch is built, loading multimodal feature data from the Paimon table in real time using a custom dataset, PaimonVectorDataset. This dataset utilizes PaimonVectorReader for incremental reading, supporting data snapshots every 2 minutes (snapshot_interval="2m"), loading only newly added or modified records to avoid full scans. During the training loop, the DataLoader inputs text and image vector features and risk labels into the model at a granularity of 1024 records per batch, and performs gradient updates using the AdamW optimizer. A model snapshot is saved and a hot update is performed every 1000 training steps, achieving an end-to-end closed loop from data ingestion to model deployment. The training cycle can be controlled within minutes, meeting the high-frequency iteration requirements of cross-border risk control scenarios. The specific implementation process is as follows:

[0202] # PyTorch Real-Time Training System

[0203] class PaimonVectorDataset(torch.utils.data.IterableDataset):

[0204] def __init__(self, table_name, snapshot_interval="2m"):

[0205] self.reader = PaimonVectorReader(

[0206] table=table_name,

[0207] vector_columns=["text_vec", "image_vec"],

[0208] batch_size=1024,

[0209] snapshot_delta=snapshot_interval )

[0211] def __iter__(self):

[0212] while True:

[0213] batch = self.reader.next_batch()

[0214] If not batch:

[0215] time.sleep(0.1) # Incremental wait

[0216] continue

[0217] yield {

[0218] "features": torch.stack(

[0219] torch.tensor(batch["text_vec"]),

[0220] torch.tensor(batch["image_vec"])

[0221] ),

[0222] "labels": torch.tensor(batch["risk_label"])

[0223] }

[0224] # Streaming training loop

[0225] model = MultimodalRiskModel()

[0226] optimizer = torch.optim.AdamW(model.parameters(), lr=1e - 4)

[0227] dataset = PaimonVectorDataset("risk_training_set")

[0228] dataloader = DataLoader(dataset, num_workers = 4)

[0229] for epoch in range(EPOCHS):

[0230] for batch in dataloader:

[0231] loss = model(batch["features"], batch["labels"])

[0232] loss.backward()

[0233] optimizer.step()

[0234] # Real-time model deployment

[0235] if step % 1000 == 0:

[0236] torch.jit.save(model, f" / models / risk_v{step}.pt")

[0237] update_serving_model(step) # Hot update inference service

[0238] The data processing method provided in this application has the following beneficial effects: 1. By designing a unified heterogeneous data storage and metadata strong association mechanism, a two-layer architecture is constructed, in which structured data is stored in the Paimon primary key table, unstructured data is stored in object storage, and strongly associated metadata records containing URIs, reserved feature vector columns, and extended metadata are registered in the Paimon table. The Paimon primary key mechanism is used to achieve unified logical management and efficient association query of heterogeneous data, and the metadata version backtracking is supported through the Time Travel capability. Efficient association query of cross-modal data is achieved, eliminating data silos, improving data governance efficiency by more than 5 times, and avoiding the latency and consistency problems of cross-system coordination in traditional solutions; 2. To simplify system component dependencies and improve the efficiency of retrieval and storage collaboration, this application introduces native deep integration of vector index and lake storage layer. The Paimon-SST format is designed by extending the Paimon storage format, embedding vector index structures such as HNSW and IVF-Flat in the SST file, realizing minute-level incremental updates of the index by listening to the feature vector column change log, extending the syntax to support native vector retrieval functions, and pushing the index calculation down to the storage layer; 3. This application establishes a Flink-based streaming feature engineering engine, deeply extending the Flink engine to introduce AI UDFs that support dynamically loaded models. It consumes data change log streams in real time, obtains unstructured data through URIs within the stream processing pipeline, calls model inference to generate feature vectors, and writes them back to the Paimon table. It adopts dynamic batch processing strategies and GPU memory pooling technology to optimize processing efficiency. The latency of unstructured data feature generation is compressed from hours to minutes or even seconds, feature freshness is improved by more than ten times, and resource utilization is improved by 25%-40%. It constructs an end-to-end real-time feature engineering pipeline with minute-level latency.

[0239] Figure 3 This is a system processing flow overview diagram of an AI model streaming collaborative training framework based on a unified lake storage architecture, according to an embodiment of this application. Figure 3 The system comprises: a data source, a unified access layer, unstructured data, a metadata registry, structured data, a vector indexing engine, streaming feature computation, AI model training, model service, and prediction result write-back. All data in the data source is first managed uniformly by the data lake storage through the unified access layer. Structured data in the data source is stored in the Paimon primary key table, while unstructured data is persistently stored using object storage (such as OSS). The metadata registry creates corresponding metadata entries in the lake storage engine (such as Apache Paimon), recording the location, feature vectors, and other extended attributes of the unstructured data through a metadata association table, achieving a strong association with the structured data. For feature vectors generated from unstructured data, the vector indexing engine is responsible for building and maintaining high-performance vector indexes, such as HNSW or IVF-Flat indexes. These indexes are directly embedded in the lake storage's SST files, providing sub-second similarity retrieval capabilities. During streaming feature computation, an extended stream processing engine (such as Apache Flink) is used to consume the change log stream of structured data and the metadata of unstructured data in real time. The AI-UDF dynamically loads the model, generating feature vectors in real time. The feature generation process is optimized through a dynamic batch processing strategy to adapt to varying data traffic intensities and ensure minimal latency. Based on feature vectors obtained from real-time streaming feature computation, combined with structured data, an AI model is trained. Model training can be either streaming training or batch training based on streaming feature generation; both utilize the latest state of the feature vectors and structured data. The trained AI model is deployed as a model service, providing online inference capabilities. The model service can receive requests, evaluate structured and unstructured data in real time, and generate prediction results. The prediction results generated by the model service are written back to the lake storage system in real time, updating the prediction fields in the structured data tables, forming a data loop. The prediction results can be used for real-time decision support, such as in financial risk control and recommendation systems.

[0240] Figure 4 This is a structural diagram of a data processing apparatus according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0241] Module 40 is used to retrieve structured data and store the structured data in the primary key table of the lake storage engine;

[0242] Create module 42 to create a metadata association table in the lake storage engine and store the description information of unstructured data in the metadata association table. The metadata association table is associated with the primary key table through the association primary key value.

[0243] The receiving module 44 is used to receive the feature vectors obtained by the stream processing engine calling the preset model to process unstructured data, and to store the feature vectors in the metadata association table.

[0244] By using the acquisition module, creation module, and receiving module in the aforementioned data processing device, the physical storage separation between structured and unstructured data is eliminated, thereby improving data processing efficiency. This solves the technical problem in related technologies where structured and unstructured data are stored in different systems, leading to complex and inefficient data processing due to physical storage separation.

[0245] The data processing device described above also includes a processing module 46, which is used to encapsulate the preset model into a user-defined function and register the user-defined function into the stream processing engine; in the stream processing engine, unstructured data stored in object storage is loaded and the unstructured data is processed by calling the user-defined function.

[0246] In the processing module of the aforementioned data processing device, the processing module is also used to capture the change log stream of the lake storage engine and obtain the capture result. The change log stream records the data changes in the lake storage engine. When the capture result indicates that the description information of unstructured data in the metadata association table has been updated, the updated description information in the change log stream is read. The updated data of unstructured data is read from the object storage based on the updated description information. The updated data is processed by calling a user-defined function.

[0247] In the processing module of the aforementioned data processing device, the processing module is also used to obtain the input flow intensity of the stream processing engine; and to adjust the batch processing parameters of the pipeline in the stream processing engine according to the input flow intensity.

[0248] In the processing module of the aforementioned data processing device, the processing module is further configured to determine the batch processing parameter as a first value when the input flow intensity is less than a first threshold; determine the batch processing parameter as a second value when the input flow intensity is greater than or equal to the first threshold and less than or equal to a second threshold, wherein the second value is greater than the first value and the second threshold is greater than the first threshold; and determine the batch processing parameter as a third value when the input flow intensity is greater than the second threshold, wherein the third value is greater than the second value.

[0249] In the processing module of the aforementioned data processing device, the processing module is also used to determine the breakpoint of the change log stream through the time travel function in the lake storage engine when the stream processing engine fails; determine the target description information to be updated after the breakpoint, and read the target update data of unstructured data from the object storage based on the target description information; and re-call the user-defined function to process the target update data.

[0250] In the processing module of the aforementioned data processing device, the processing module is further configured to: obtain model version information from the metadata association table when the preset model is updated from the first version to the second version; determine the first feature vector generated by the preset model of the first version based on the model version information; and call a user-defined function to process the unstructured data corresponding to the first feature vector using the preset model of the second version.

[0251] In the processing module of the aforementioned data processing device, the feature vector includes text vectors and image vectors. This processing module is also used to obtain predefined query vectors, where the query vectors include query text and query image; determine the account ID from the metadata association table, and retrieve the target text vector and target image vector of the account ID using a vector search function to determine the first similarity between the target text vector and the query text, and the second similarity between the target image vector and the query image, where the account ID is the associated primary key value; determine the account balance in the primary key table based on the account ID in the metadata association table; identify account IDs with account balances exceeding a preset value as target account IDs; determine the target first similarity and target second similarity corresponding to the target account IDs, and perform weighted fusion of the target first similarity and target second similarity to obtain a comprehensive score; identify the account IDs with the highest comprehensive scores (within a preset number of times) as risk account IDs.

[0252] It should be noted that, Figure 4 The data processing device shown is used to perform Figure 2 The data processing method shown above is also applicable to this data processing device, and will not be repeated here.

[0253] This application also provides an electronic device, which includes a memory and a processor. The memory stores program instructions, and the processor is connected to the memory to execute program instructions that perform the following functions: acquiring structured data and storing the structured data in a primary key table in a lake storage engine; creating a metadata association table in the lake storage engine and storing description information of unstructured data in the metadata association table, wherein the metadata association table is associated with the primary key table through association primary key values; receiving feature vectors obtained by a stream processing engine processing unstructured data using a preset model, and storing the feature vectors in the metadata association table.

[0254] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The data processing method shown above is also applicable to this electronic device, and will not be repeated here.

[0255] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following data processing method by running the computer program: acquiring structured data and storing the structured data in a primary key table in a lake storage engine; creating a metadata association table in the lake storage engine and storing the description information of the unstructured data in the metadata association table, wherein the metadata association table is associated with the primary key table through a primary key value; receiving a feature vector obtained by a stream processing engine processing the unstructured data using a preset model, and storing the feature vector in the metadata association table.

[0256] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The data processing method shown above is also applicable to this non-volatile storage medium, and will not be repeated here.

[0257] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the data processing methods in various embodiments of this application.

[0258] This application also provides a computer program that, when executed by a processor, implements the steps of the data processing methods in various embodiments of this application.

[0259] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0260] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0261] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0262] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0263] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0264] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0265] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / float> < / float> < / float> < / float>

Claims

1. A data processing method, characterized in that, include: Obtain structured data and store the structured data in the primary key table of the lake storage engine; A metadata association table is created in the lake storage engine, and the description information of unstructured data is stored in the metadata association table. The metadata association table is associated with the primary key table through the association primary key value. The receiving stream processing engine calls a preset model to process the unstructured data to obtain a feature vector, and stores the feature vector in the metadata association table.

2. The method according to claim 1, characterized in that, The stream processing engine calls a preset model to process the unstructured data, including: The preset model is encapsulated as a user-defined function, and the user-defined function is registered to the stream processing engine; In the stream processing engine, unstructured data stored in object storage is loaded, and the unstructured data is processed by calling the user-defined function.

3. The method according to claim 2, characterized in that, The stream processing engine calls a preset model to process the unstructured data, including: Capture the change log stream of the lake storage engine to obtain the capture result, wherein the change log stream records the data changes in the lake storage engine; If the capture result indicates that the description information of unstructured data in the metadata association table has been updated, read the updated description information from the change log stream; Updated data of unstructured data is read from object storage based on the update description information; The user-defined function is invoked to process the updated data.

4. The method according to claim 2, characterized in that, The method further includes: Obtain the input flow intensity of the stream processing engine; The batch processing parameters of the pipeline in the stream processing engine are adjusted according to the input flow intensity.

5. The method according to claim 4, characterized in that, Adjusting the batch processing parameters of the pipeline in the stream processing engine based on the input flow intensity includes: If the input flow intensity is less than a first threshold, the batch processing parameter is determined to be a first value; If the input flow intensity is greater than or equal to the first threshold and less than or equal to the second threshold, the batch processing parameter is determined to be a second value, wherein the second value is greater than the first value and the second threshold is greater than the first threshold; If the input flow intensity is greater than the second threshold, the batch processing parameter is determined to be a third value, wherein the third value is greater than the second value.

6. The method according to claim 3, characterized in that, The method further includes: In the event of a failure in the stream processing engine, the breakpoint of the change log stream is determined using the time travel function in the lake storage engine. After determining the breakpoint, update the target description information, and read the target update data of unstructured data from the object storage based on the target description information; The user-defined function is invoked again to process the target update data.

7. The method according to claim 2, characterized in that, The method further includes: When the preset model is updated from version 1 to version 2, the model version information in the metadata association table is obtained; Based on the model version information, determine the first feature vector generated by the preset model of the first version; The user-defined function is invoked to process the unstructured data corresponding to the first feature vector using the second version of the preset model.

8. The method according to claim 1, characterized in that, The feature vector includes text vectors and image vectors, and the method further includes: Obtain a predefined query vector, wherein the query vector includes query text and query image; The account ID is determined from the metadata association table, and the target text vector and target image vector of the account ID are retrieved using a vector search function to determine the first similarity between the target text vector and the query text, and the second similarity between the target image vector and the query image, wherein the account ID is the association primary key value; Based on the account ID in the metadata association table, determine the account balance in the primary key table; The account ID whose account balance exceeds a preset value is identified as the target account ID; Determine the first similarity and second similarity of the target account ID, and perform a weighted fusion of the first similarity and the second similarity to obtain a comprehensive score; The top-ranked account IDs with the highest comprehensive scores are identified as risk account IDs.

9. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire structured data and store the structured data in the primary key table of the lake storage engine; A creation module is used to create a metadata association table in the lake storage engine and store the description information of unstructured data in the metadata association table, wherein the metadata association table is associated with the primary key table through the association primary key value; The receiving module is used to receive the feature vector obtained by the stream processing engine calling the preset model to process the unstructured data, and to store the feature vector in the metadata association table.

10. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor, connected to the memory, is configured to execute program instructions that perform the following functions: acquire structured data and store the structured data in a primary key table in the lake memory engine; A metadata association table is created in the lake storage engine, and the description information of unstructured data is stored in the metadata association table. The metadata association table is associated with the primary key table through the association primary key value. The receiving stream processing engine calls a preset model to process the unstructured data to obtain a feature vector, and stores the feature vector in the metadata association table.

11. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the data processing method according to any one of claims 1 to 8 by running the computer program.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data processing method according to any one of claims 1 to 8.

Images