Data processing method and device
By combining the target distributed computing framework with a columnar storage system, the problems of inconsistent and chaotic multimodal data storage and management are solved, achieving efficient data processing and resource utilization, and reducing network transmission overhead and computational result deviation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI BILIBILI TECH CO LTD
- Filing Date
- 2026-01-28
- Publication Date
- 2026-05-12
AI Technical Summary
Multimodal data storage in different storage systems leads to data inconsistency and management chaos. Furthermore, in distributed computing, retrieving data across storage media incurs network transmission overhead and results in deviations, resulting in low efficiency.
By combining a target distributed computing framework with a columnar storage system, multimodal data is updated to the target column through streaming and batch processing, and processing results are read from and written to the columnar storage system when needed, and computational operations are performed using a data processing pipeline.
It enables unified storage and efficient management of multimodal data, reduces data inconsistency and management chaos, lowers network transmission overhead and calculation result deviation, and improves data usage and resource utilization efficiency.
Smart Images
Figure CN122019162A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed computing technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology
[0002] With the development of large model technology, many business operations related to large models often generate a large amount of multimodal data. Since these multimodal data have different formats, they are usually stored in different storage systems.
[0003] However, storing multimodal data in different storage systems can easily lead to data inconsistency and chaotic data management. If this multimodal data is needed later, it needs to be retrieved from multiple storage media, which is inefficient.
[0004] On the other hand, since large models typically require large-scale computations, and single-point computation has limited performance, distributed computing is necessary to meet these requirements. However, using distributed computing involves significant network transmission overhead due to multiple nodes retrieving data across storage media, and inconsistent data versions across different nodes can easily lead to deviations in computation results. These problems, such as data inconsistency, chaotic data management, and low retrieval efficiency, become even more pronounced.
[0005] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention
[0006] This application provides a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems mentioned above.
[0007] One aspect of this application provides a data processing method, the method comprising:
[0008] In the case where multimodal data is generated by processing using a target distributed computing framework and the multimodal data already corresponds to a target column of a columnar storage system, the multimodal data is updated to the target column. The columnar storage system supports the storage of heterogeneous data, and the target distributed computing framework supports streaming processing and batch processing. If the multimodal data does not correspond to any column of the columnar storage system, the multimodal data is stored in a newly added column of the columnar storage system. In cases where target multimodal data needs to be acquired, the target distributed computing framework is used to read the target multimodal data from the columnar storage system, the target multimodal data is processed, and the result of the processing is written into the columnar storage system.
[0009] Optionally, when it is necessary to obtain target multimodal data, reading the target multimodal data from the columnar storage system using a target distributed computing framework, processing the target multimodal data, and writing the processed result of the target multimodal data into the columnar storage system includes: In cases where it is necessary to obtain target multimodal data, the target distributed computing framework is used to read target data fragments from the columnar storage system. The target data fragment is one of the fragments of the target multimodal data, and the target data fragment includes a target fragment ID. The target distributed computing framework is used to perform target computing operations on the target data shards according to the target shard ID to obtain the operation results. The operation results are merged based on the target shard ID and then written into the columnar storage system.
[0010] Optionally, the step of using the target distributed computing framework to perform target computation operations on the target data shards according to the target shard ID to obtain the operation result includes: The first data of the target data fragments is converted from column format to second data of dictionary data structure using the target distributed computing framework. The target distributed computing framework is used to perform target computation operations on the second data based on the target shard ID to obtain the operation result. Correspondingly, the step of merging the operation results based on the target shard ID and writing them into the columnar storage system includes: The target distributed computing framework is used to convert the operation results back into third data in column format; The third data is merged based on the target shard ID and then written into the columnar storage system.
[0011] Optionally, the step of using the target distributed computing framework to perform target computation operations on the second data based on the target shard ID includes: The target distributed computing framework is used to invoke a pre-configured data processing pipeline, wherein the data processing pipeline is configured using a data serialization language. The data processing pipeline performs target calculation operations on the second data based on the target shard ID.
[0012] Optionally, the data processing pipeline includes standardized operators, which include general operators and custom operators. The general operators are operators pre-built in the target distributed computing framework, and the custom operators are operators configured using the operator standardization interface in the target distributed computing framework. The configuration of the operator standardization interface includes operator initialization configuration, operator input, operator processing logic, and operator output. The operator processing logic includes single-line processing and batch processing. Correspondingly, the step of utilizing the target distributed computing framework to invoke a pre-configured data processing pipeline, and performing target computation operations on the second data according to the target shard ID through the data processing pipeline, includes: Obtain the configuration of the standardized operator; The standardized operator is invoked based on its configuration. The standardized operator performs target calculation operations on the second data based on the target fragment ID.
[0013] Optionally, the configuration of the data processing pipeline includes execution engine type, engine execution mode, data processing mode, data input configuration, operator concatenation configuration, and data output configuration. The engine execution mode includes cluster mode and local mode, and the data processing mode includes batch processing mode and stream processing mode. Correspondingly, the step of using the target distributed computing framework to invoke a pre-configured data processing pipeline includes: Obtain the configuration of the data pipeline; The target distributed computing framework invokes the data pipeline based on the configuration of the data pipeline.
[0014] Optionally, when the data processing mode is the stream processing mode, the configuration of the data processing pipeline further includes breakpoint resume, elastic scaling, queue capacity control, and block size control.
[0015] Optionally, reading target data shards from the columnar storage system using the target distributed computing framework includes: The target data fragments are pre-read from the columnar storage system using a data loader; The target data is sharded and cached in the distributed object storage of the target distributed computing framework; The data loader is used to pull the target data fragment from the distributed object storage according to a preset data reading mode and populate it into the local cache queue; The target data fragment is obtained from the cache queue using the target distributed computing framework.
[0016] Optionally, the configuration of the data loader includes the read path from the columnar storage system, the field columns to be read, the data filtering conditions, the total number of samples to be read, the distributed training configuration, the batch data to be read beforehand, the data reading mode, the data transformation, and the hardware adaptation. The data reading mode includes asynchronous reading and streaming reading. Correspondingly, the step of pre-reading target data fragments from the columnar storage system using a data loader includes: Obtain the configuration of the data loader, and call the data loader to pre-read target data fragments from the columnar storage system according to the configuration of the data loader; The step of using the data loader to pull the target data shard from the distributed object storage according to a preset data reading mode includes: The data reading mode is determined according to the configuration of the data loader, and the target data fragment is pulled from the distributed object storage according to the configured data reading mode.
[0017] Another aspect of this application provides a data processing apparatus, the apparatus comprising: An update module is used to update the multimodal data to the target column when the multimodal data generated by the target distributed computing framework is processed and the multimodal data already corresponds to the target column of the columnar storage system. The columnar storage system supports the storage of heterogeneous data, and the target distributed computing framework supports streaming processing and batch processing. A storage module is used to store the multimodal data into a newly added column of the columnar storage system when the multimodal data does not correspond to any column of the columnar storage system. The processing module is used to read the target multimodal data from the columnar storage system using the target distributed computing framework when it is necessary to obtain the target multimodal data, process the target multimodal data, and write the processed result of the target multimodal data into the columnar storage system.
[0018] Another aspect of this application provides a computer device, including: At least one processor; and A memory that is communicatively connected to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.
[0019] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.
[0020] Another aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described above.
[0021] The embodiments of this application employing the above-described technical solution may have the following advantages: By utilizing the target distributed computing framework to process multimodal data, and when the multimodal data already corresponds to a target column in the columnar storage system, the multimodal data is updated to the target column, and the columnar storage system supports the storage of heterogeneous data. When the multimodal data does not correspond to any column in the columnar storage system, the multimodal data is stored in a newly added column in the columnar storage system. When it is necessary to retrieve the target multimodal data, the distributed computing framework is used to read the target multimodal data from the columnar storage system, and the processed result is then written back to the columnar storage system. This design, combining the target distributed computing framework with the columnar storage system, can meet the distributed computing needs of large-scale business applications. Simultaneously, storing multimodal data uniformly in the columnar storage system reduces problems such as data inconsistency, management chaos, and low retrieval efficiency caused by storing multimodal data in different storage media. It also reduces network transmission overhead and computational result deviations when retrieving data across storage media, improving data utilization efficiency and resource efficiency. Attached Figure Description
[0022] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0023] Figure 1 This diagram schematically illustrates the operating environment of the data processing method according to Embodiment 1 of this application; Figure 2 The schematic diagram illustrates the architecture of the target distributed computing framework; Figure 3 A flowchart illustrating a data processing method according to Embodiment 1 of this application is shown schematically. Figure 4 The illustration shows a new flow in the data processing method according to Embodiment 1 of this application; Figure 5 This illustration shows an example diagram of slice calculation performed according to the data processing method of Embodiment 1 of this application; Figure 6 Schematic illustration Figure 3 Flowcharts of the sub-steps in steps S302 and S304; Figure 7 Schematic illustration Figure 6 Flowchart of the sub-steps in step S402; Figure 8 Schematic illustration Figure 7 Flowcharts of the sub-steps in steps S500 and S502; Figure 9 Schematic illustration Figure 7 Flowchart of another sub-step in step S500; Figure 10 Schematic illustration Figure 4 Flowchart of the sub-steps in step S300; Figure 11 An example diagram illustrating the use of a data loader to load data in the data processing method according to Embodiment 1 of this application is shown. Figure 12 This schematic diagram illustrates the overall architecture of the data processing method according to Embodiment 1 of this application; Figure 13 A block diagram of a data processing apparatus according to Embodiment 2 of this application is schematically shown; and Figure 14 A schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of this application is shown. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0025] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0026] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.
[0027] First, a definition of the terminology used in this application is provided: Dictionary data structure: It is an unstructured / semi-structured way of organizing data. Each data unit consists of a unique identifier key and a corresponding associated value. The key is unique and is used to quickly index the corresponding value. The value can carry multiple types of data (including strings, numbers, arrays, binary streams, and even nested key-value pairs).
[0028] Operator: An operator is a functional encapsulation that performs a single, specific operation on data. It takes raw / intermediate data as input and outputs processed data. It has no extra redundant logic and has the characteristics of "high cohesion and low coupling".
[0029] Object storage is a technology that uses "objects" as the basic unit for storing and managing data. Each object consists of a data ontology, extended metadata, and a unique identifier, and all objects reside in buckets within a flat namespace. Object storage offers near-infinite horizontal scalability, making it suitable for the long-term storage and on-demand access of massive amounts of diverse unstructured data.
[0030] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below: In related technologies, the large amounts of multimodal data generated by large-scale model-related business operations are typically stored in different storage systems, which can easily lead to data inconsistency, chaotic data management, and low retrieval efficiency. On the other hand, large-scale model-related business operations usually require distributed computing to meet the requirements. When using distributed computing, retrieving data across storage media by multiple nodes generates significant network transmission overhead, and inconsistent data versions on different nodes can easily lead to deviations in calculation results. The aforementioned problems of data inconsistency, chaotic data management, and low retrieval efficiency become even more prominent.
[0031] Therefore, this application provides a data processing technical solution. In this solution, by designing a target distributed computing framework plus a columnar storage system, the target distributed computing framework meets the computational requirements of large-scale business applications. Simultaneously, the generated multimodal data is uniformly stored in the columnar storage system. This reduces problems such as data inconsistency, management chaos, and low retrieval efficiency caused by storing multimodal data on different storage media. It also reduces network transmission overhead and computational result deviations when retrieving data across storage media, thereby improving data utilization efficiency and resource efficiency. See below for details.
[0032] Finally, for ease of understanding, an exemplary operating environment is provided below.
[0033] like Figure 1 As shown, the operating environment diagram includes a columnar storage system 10 and a target distributed computing framework 20. The target distributed computing framework 20 can read multimodal data from the columnar storage system 10, and then use the multimodal data for distributed computing. After the distributed computing is completed, the target distributed computing framework 20 can store the computed multimodal data back into the columnar storage system 10. It should be noted that the target distributed computing framework 20 can also read data from other data sources. After processing, in addition to storing the data in the columnar storage system, the target distributed computing framework 20 can also store some of the data in other locations.
[0034] Figure 2 The diagram illustrates the general architecture of the target distributed computing framework, as shown in the figure. It mainly includes a three-layer design, with each layer having clearly defined responsibilities and working collaboratively to support the efficient execution of distributed computing tasks. 1. Application Layer: This is the entry point for users to interact with the target distributed computing framework, and mainly includes three types of components: Tasks / Jobs: Distributed computing tasks submitted by users (such as model training and data processing jobs); Services / APIs: Capabilities provided to the outside world in the form of services (such as real-time inference APIs and distributed service interfaces). Library functions: Tool libraries provided by the target distributed computing framework.
[0035] 2. Core Layer (Cluster Logic Layer) of the Target Distributed Computing Framework: This is the "brain and execution center" of the target distributed computing framework, mainly comprising three types of core components: Head node (control center): The control node of the target distributed computing framework cluster, responsible for resource scheduling, task allocation, and metadata management (such as tracking the status of execution units), is the "command center" of the cluster; Work unit node (computing node): The "computing power carrier" of the cluster, which carries the actual execution of the execution unit and can be elastically scaled up or down according to the load; Execution unit: The smallest execution entity in the target distributed computing framework, which can include stateful and stateless tasks, and releases resources after execution.
[0036] 3. Infrastructure Layer: This is the "physical / virtual resource foundation" upon which the target distributed computing framework runs, providing three types of basic support: Computing resources (CPU / GPU): provide computing power for work units and execution units, and support heterogeneous computing (such as simultaneously scheduling CPU tasks and GPU accelerated tasks). Storage resources (local / distributed): Provides data storage capabilities, supporting both temporary storage on local disks and compatibility with distributed storage systems such as HDFS and cloud object storage; Network resources: Ensure communication between nodes within the cluster.
[0037] The technical solutions of this application are described below through several embodiments. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein.
[0038] Example 1 Figure 3 A flowchart illustrating a data processing method according to Embodiment 1 of this application is shown schematically. It should be noted that the executing entity of the data processing method in this embodiment can be a target distributed computing framework, wherein the target distributed computing framework can be a Ray architecture or a similar architecture. Figure 2 A similar architecture is shown. Furthermore, columnar storage systems support compatibility with heterogeneous data, allowing multimodal data to be stored uniformly by field column. They support dynamic addition and updating of columns, independent read, write, append, and update operations on target columns, sharding and indexing capabilities, and provide access interfaces compatible with target distributed computing frameworks, supporting cross-node data sharding reading and result merging writing. Optionally, the columnar storage system can be Lance or a columnar storage system that supports the above capabilities.
[0039] like Figure 3 As shown, the data processing method may include steps S200~S204, wherein: Step S200: When multimodal data is generated by processing using the target distributed computing framework and the multimodal data already corresponds to the target column of the columnar storage system, the multimodal data is updated to the target column. The columnar storage system supports the storage of heterogeneous data, and the target distributed computing framework supports streaming processing and batch processing.
[0040] Step S202: If the multimodal data does not correspond to any column of the columnar storage system, store the multimodal data in a newly added column of the columnar storage system.
[0041] Step S204: When it is necessary to obtain target multimodal data, the target multimodal data is read from the columnar storage system using the target distributed computing framework, the target multimodal data is processed, and the result of the target multimodal data processing is written into the columnar storage system.
[0042] The data processing method provided in this embodiment updates the multimodal data to the target column when the multimodal data already corresponds to a target column in the columnar storage system, as the target distributed computing framework processes the generated multimodal data. The columnar storage system supports the storage of heterogeneous data, and the target distributed computing framework supports streaming and batch processing. When the multimodal data does not correspond to any column in the columnar storage system, the multimodal data is stored in a newly added column in the columnar storage system. When it is necessary to retrieve the target multimodal data, the distributed computing framework is used to read the target multimodal data from the columnar storage system, and the processed result is then written back to the columnar storage system. Through the base design of the target distributed computing framework plus the columnar storage system, the distributed computing needs of large-scale business can be met. At the same time, storing multimodal data uniformly in the columnar storage system reduces problems such as data inconsistency, management chaos, and low retrieval efficiency caused by storing multimodal data in different storage media. It also reduces network transmission overhead and calculation result deviation when retrieving data across storage media, thereby improving data utilization efficiency and resource utilization efficiency.
[0043] The following combination Figure 3 The steps in steps S200 to S204, as well as other optional steps, are described in detail.
[0044] Step S200 When multimodal data is generated by the target distributed computing framework and the multimodal data already corresponds to the target column of the columnar storage system, the multimodal data is updated to the target column. The columnar storage system supports the storage of heterogeneous data, and the target distributed computing framework supports streaming and batch processing.
[0045] Multimodal data is a type of heterogeneous data, which can include video data, audio data, image data, text data, and vector data, among others.
[0046] The target distributed computing framework can acquire and process data from a columnar storage system or other data sources. It then determines whether the resulting multimodal data corresponds to a target column in the columnar storage system. If they do, the multimodal data is updated in the target column. The target column can be any column in the columnar storage system. Optionally, there can be multiple target columns, and correspondingly, multiple sets of multimodal data. In this case, each set of multimodal data is updated in its corresponding target column in the columnar storage system. Furthermore, if the multimodal data consists of newly added complete multimodal records, requiring the addition of new rows to the existing column structure of the columnar storage system, the multimodal data is appended to the new row in the target column according to the field format of the corresponding column, achieving row-level expansion of the multimodal data.
[0047] Step S202In cases where multimodal data does not correspond to any column in the columnar storage system, the multimodal data is stored in a newly added column of the columnar storage system.
[0048] Specifically, after processing, the target distributed computing framework makes a judgment. If the generated multimodal data does not correspond to any column of the columnar storage system, a new column can be added to the columnar storage system, and then the multimodal data can be stored in the newly added column.
[0049] Step S204 In situations where target multimodal data needs to be obtained, the target distributed computing framework is used to read the target multimodal data from the columnar storage system, process the target multimodal data, and then write the processed result into the columnar storage system.
[0050] Specifically, when it is necessary to obtain target multimodal data for computation, the target distributed computing framework can be used to read the target multimodal data from the columnar storage system. After processing the target multimodal data, the result of the processing is written into the columnar storage system.
[0051] In optional embodiments, such as Figure 4 As shown, step S204 may further include: Step S300: When it is necessary to obtain target multimodal data, the target data fragment is read from the columnar storage system using the target distributed computing framework. The target data fragment is one of the fragments of the target multimodal data and includes the target fragment ID.
[0052] Step S302: Using the target distributed computing framework, target data is sharded according to the target shard ID to perform target computing operations and obtain the operation results.
[0053] Step S304: The operation results are merged based on the target shard ID and written to the columnar storage system.
[0054] Specifically, a columnar storage system internally organizes multimodal data into multiple data shards. Each shard has a unique shard ID and records its metadata, such as storage path, data row range, and column information. When a task starts, the target distributed computing framework can calculate the target shard ID (corresponding to the target data shard) to be processed by the current node using a sharding algorithm (such as hash modulo) based on the total data size, node data, and the current node sequence number. Then, the target distributed computing framework can send a request to the columnar storage system to read the target data shard. After reading the target data shard, a pre-configured data processing pipeline is invoked to process the target data shard. This pipeline can include one or more operators. Throughout the computation, the target shard ID is retained and transmitted. After the computation is complete, the operation result is obtained. The target distributed computing framework can merge the operation result based on the target shard ID and then write it to the columnar storage system.
[0055] Figure 5 The diagram illustrates an example of sharded computation. As shown, multiple data shards (A / B, C, D) are first read from the "column-based data source." Then, in the "data processing pipeline" stage, each shard is split and assigned to a corresponding row-level task for parallel processing. After processing, the write results for the corresponding shards are generated (write shards A, B, C, D). Finally, all write shards are aggregated and submitted through the "merge" stage to form the complete processed data (submit shards A / B / C / D).
[0056] In this embodiment, by using a target distributed computing framework to read target data fragments from a columnar storage system, and then using the target distributed computing framework to perform target computation operations on the target data fragments according to the target fragment ID, the operation results are obtained. Based on the target fragment ID, the operation results are merged and written into the columnar storage system. This allows large volumes of target multimodal data to be split into multiple fragments for parallel processing, avoiding the performance bottleneck of single-node processing and shortening data processing time. At the same time, by using fragment IDs for operations, abnormal fragments can be quickly located, and the problems of multimodal data being disordered or lost in distributed processing can also be avoided.
[0057] In optional embodiments, such as Figure 6 As shown, in step S302, the target distributed computing framework is used to perform target computation operations on the target data shards according to the target shard ID to obtain the operation results, which may further include steps S400 to S402; correspondingly, in step S304, the operation results are merged based on the target shard ID and written to the columnar storage system, which may further include steps S404 to S406, wherein: Step S400: Using the target distributed computing framework, the first data of the target data shards is converted from column format to the second data of dictionary data structure.
[0058] Step S402: Use the target distributed computing framework to perform target computation operations on the second data according to the target shard ID to obtain the operation result.
[0059] Step S404: Use the target distributed computing framework to convert the operation result back into third data in column format.
[0060] Step S406: The third data is merged based on the target shard ID and then written into the columnar storage system.
[0061] Specifically, after the target distributed computing framework reads the target data shards from the columnar storage system, it maps the first data in column format to the second data in a dictionary data structure of key-value pairs. The target distributed computing framework can use the target shard ID as an identifier to assign the second data of the dictionary data structure to the operators of the data processing pipeline for target computation operations. After the target computation operation is completed, the operation result is obtained. The target distributed computing framework remaps the operation result of the dictionary data structure to the third data in column format corresponding to the columnar storage system. Then, using the target shard ID as a unified identifier, the third data in column format corresponding to each data shard is aligned and merged, and finally the merged complete column data is written into the corresponding column of the columnar storage system.
[0062] In this embodiment, the read column-formatted data is converted into dictionary data format for processing, and the operation result is then converted back into column format for storage. This allows the operator to be "unaware" of the underlying storage format through the dictionary data structure, enabling operator developers to use a familiar dictionary data structure to develop operators, thus lowering the threshold for operator development and achieving both efficient column-formatted storage and flexible operator processing.
[0063] In an optional embodiment, in step S402, the target distributed computing framework is used to perform target computation operations on the second data based on the target shard ID, such as... Figure 7 As shown, it may further include: Step S500: The target distributed computing framework is used to call a pre-configured data processing pipeline, which is configured using a data serialization language.
[0064] Step S502: Perform target calculation operation on the second data according to the target fragment ID through the data processing pipeline.
[0065] Specifically, the data processing workflow (such as frame extraction, feature extraction, image quality evaluation, and video noise reduction) can be predefined using a data serialization language (such as Yaml) to obtain a data processing pipeline. After converting the first data from column format to a dictionary data structure, the target distributed computing framework can call the data processing pipeline to perform target computation operations on the second data according to the predefined processing flow of the data processing pipeline. The workflow logic in the data processing pipeline can correspond to a DAG (Directed Acyclic Graph), for example, the workflow logic of the data processing pipeline can be defined based on each node of the DAG.
[0066] In this embodiment, a pre-configured data processing pipeline is invoked using the target distributed computing framework. The data processing pipeline is configured using a data serialization language. The data processing pipeline performs target computation operations on the second data based on the target shard ID. The data processing pipeline can be configured using a data serialization language, which reduces the development difficulty of the data processing pipeline. At the same time, the data processing pipeline can split the task into multiple nodes in the cluster for processing, thereby improving the efficiency of task processing.
[0067] In a data processing pipeline, several operators can be used to perform target calculation operations on the second data.
[0068] In an optional embodiment, the data processing pipeline may include standardized operators, which include general operators and custom operators. The general operators are operators pre-built in the target distributed computing framework, and the custom operators are operators configured using the operator standardization interface in the target distributed computing framework. The configuration of the operator standardization interface includes operator initialization configuration, operator input, operator processing logic, and operator output. The operator processing logic includes single-line processing and batch processing. Correspondingly, such as Figure 8 As shown, steps S500 and S502 may further include: Step S600: Obtain the configuration of the standardized operator.
[0069] Step S602: Invoke the standardized operator based on the configuration of the standardized operator.
[0070] Step S604: Perform target calculation operation on the second data based on the target fragment ID using a normalization operator.
[0071] The general operators can be pre-developed operators for commonly used operations (such as frame extraction, OCR recognition, and feature extraction), which can be used directly. Specifically, they can be developed through code or through a standardized operator interface. The configuration of the standardized operator interface includes: operator initialization configuration (such as the model path and processing parameters, like the frame extraction interval); operator input configuration (such as the data input path); operator processing logic configuration (such as the specific operations performed by the operator, like frame extraction); and operator output configuration (such as the output location). The operator processing logic can be for fine-grained processing of a single data point (such as fine-grained OCR recognition of a single video) or batch processing (processing multiple data points at once), which can be implemented using a batch processing list.
[0072] Specifically, when calling the data processing pipeline, the configuration of the standardized operator can be obtained, the corresponding standardized operator can be called according to the configuration, and then the target calculation operation can be performed on the second data according to the target shard ID through the called standardized operator.
[0073] In this embodiment, by obtaining the configuration of the standardized operator, calling the standardized operator according to the configuration, and performing target calculation operations on the second data according to the target shard ID, since the general operator can be called directly, repeated development can be avoided and the cost of use can be reduced; while custom operators can be obtained through the operator standardization interface configuration, which can reduce the difficulty of development and improve the development efficiency of operators compared to developing from scratch.
[0074] In optional embodiments, the configuration of the data processing pipeline may include execution engine type, engine execution mode, data processing mode, data input configuration, operator concatenation configuration, and data output configuration. The engine execution mode may include cluster mode and local mode, and the data processing mode may include batch processing mode and stream processing mode. Correspondingly, such as Figure 9 As shown, step S500 may further include: Step S700: Obtain the configuration of the data pipeline.
[0075] Step S702: Utilize the target distributed computing framework to invoke the data pipeline based on the data pipeline configuration.
[0076] The execution engine type specifies whether it is a target distributed computing engine. Engine execution modes can include cluster mode and local mode. Cluster mode is suitable for scenarios requiring cluster computing resources, such as big data or high concurrency, while local mode runs only locally and is suitable for debugging. Data processing modes include batch mode and stream processing mode. Batch mode is suitable for processing a batch of data at once and is suitable for offline tasks, such as historical data backtracking. Stream processing mode is suitable for reading incremental data and is suitable for near real-time tasks, such as processing uploaded videos immediately. Data input configuration defines the data source read path, read concurrency, etc. The read path must at least include the read path of the columnar storage system and may also include read paths of other data sources. Operator concatenation configuration refers to the relationship or execution order between operators. For example, they can be executed sequentially in list order, with the output of one operator becoming the input of the next. Data output configuration defines the output path of the final result. The output path must at least include the location of the columnar storage system and may also include other data storage locations.
[0077] Specifically, when calling the data pipeline, you can first obtain the various configurations of the data pipeline, and then call the data pipeline to process the second data according to the various configurations of the data pipeline.
[0078] In this embodiment, the configuration of the data processing pipeline can achieve rapid debugging through local mode without relying on the complete cluster, thus improving debugging efficiency. The configuration of batch processing mode and stream processing mode enables the target distributed computing framework to support both batch and stream processing data processing modes, improving the flexibility of data processing. In addition, the configuration of the data processing pipeline can meet the data link requirements of different businesses.
[0079] In an optional embodiment, when the data processing mode is stream processing mode, the configuration of the data processing pipeline also includes breakpoint resume, elastic scaling, queue capacity control, and block size control.
[0080] Among these features, breakpoint resumption can be configured to persistently store processing status at regular intervals, allowing for job recovery based on the breakpoint resumption configuration after an abnormal interruption. Elastic scaling allows for the configuration of scaling rules, specifying under what conditions scaling up or down; for example, scaling down during a specific time period at night (when business volume is low) can save costs. Queue capacity control allows for the configuration of the maximum queue capacity, such as the maximum queue length, to avoid memory overflow or data backlog, maintaining system stability and real-time performance. Block size control allows for the configuration of the size of each data block in stream processing; for example, the block size can be configured according to time windows, allowing for appropriately increasing the block size in high-throughput scenarios to improve efficiency, and appropriately decreasing the block size in low-latency scenarios to shorten processing latency.
[0081] In an optional embodiment, in step S300, the target data fragments are read from the columnar storage system using the target distributed computing framework, such as... Figure 10 As shown, it may further include: Step S800: Use a data loader to pre-read the target data fragments from the columnar storage system.
[0082] Step S802: The target data is sharded and cached in the distributed object storage of the target distributed computing framework.
[0083] Step S804: Use the data loader to pull the target data fragment from the distributed object storage according to the preset data reading mode and fill it into the local cache queue.
[0084] Step S806: Use the target distributed computing framework to obtain the target data fragment from the cache queue.
[0085] The data loader can be a component that bridges the columnar storage system and the target distributed computing framework. It can be implemented by defining various reading parameters of the data loader. It only needs to be able to pre-read the target data fragments from the columnar storage system, cache the target data fragments in the distributed object storage of the target distributed computing framework, pull the target data fragments from the distributed object storage and populate them into the local cache queue.
[0086] Specifically, a data loader can be used to read target data fragments from a columnar storage system according to a specific configuration; then, the read fragments are cached in the distributed object storage built into the target distributed computing framework, allowing data to be transferred from external storage to the internal storage layer of the target distributed computing framework, thus shortening the physical distance between data and computing resources; the data loader can then use a preset data reading mode (such as batch reading) to pull target data fragments from the distributed object storage and fill them into the local cache queue; when the target distributed computing framework processes the target data fragments on the demand side, it retrieves the target data fragments from the local cache queue.
[0087] Please refer to Figure 11 This is an example diagram of using a data loader to preload data. As shown in the figure, the data loader asynchronously reads data from the columnar storage system. After sampling, tensor transformation, and preprocessing, it outputs standardized training data through data organization. The processed data is passed to the network object storage through the GPU trainer, and then asynchronously retrieved and fed to the trainer thread by the data loader. The trainer thread simultaneously performs model forward propagation and backward propagation, and finally the CUDA thread calls the kernel to complete the GPU-level computational acceleration.
[0088] In this embodiment, by using a data loader to pre-read target data fragments from the columnar storage system and cache them in the distributed object storage of the target distributed computing framework, the target data fragments are pulled from the distributed object storage according to a preset data reading mode and filled into the local cache queue. Finally, the target distributed computing framework obtains the target data fragments from the local cache queue. The pre-loading by the data loader can significantly shorten the time for the target distributed computing framework to obtain data, thereby improving the computing efficiency and resource utilization of the target distributed computing framework.
[0089] In an optional embodiment, the data loader can be implemented through configuration. The configuration of the data loader may include the read path from the columnar storage system, the field columns to be read, the data filtering conditions, the total number of samples to be read, the distributed training configuration, the batch data to be read before reading, the data reading mode, data transformation and hardware adaptation. The data reading mode includes asynchronous reading and streaming reading. Correspondingly, in step S800, the process of pre-reading the target data fragment from the columnar storage system using the data loader may include: obtaining the configuration of the data loader, and calling the data loader to pre-read the target data fragment from the columnar storage system according to the configuration of the data loader; In step S804, the data loader pulls the target data shard from the distributed object storage according to the preset data reading mode, which may include: determining the data reading mode according to the configuration of the data loader, and pulling the target data shard from the distributed object storage according to the configured data reading mode.
[0090] The "read fields" section refers to the columns retrieved from the columnar storage system. Data filtering conditions configure which types of data to filter, such as filtering samples with video durations less than 30 seconds. Distributed training configuration specifies the data for each node during distributed training, such as configuring data splitting by batch. Pre-fetched batch data refers to pre-fetched batches of data, such as pre-fetching two batches. Data reading modes include asynchronous reading and streaming reading. Asynchronous reading, for example, uses Ray's Actor to asynchronously read data, suitable for large-scale data scenarios; streaming reading, for example, uses Ray's Pipeline to stream data, suitable for streaming training scenarios. Data transformation can convert the columnar storage system's format to a dictionary data structure, thereby converting heterogeneous data into a format that the model can process. Hardware adaptation configuration can include whether to load data using the GPU, suitable for scenarios where data is directly read to the GPU to reduce CPU-to-GPU copy time.
[0091] In this embodiment, by configuring the data loader accordingly, data can be moved from the columnar storage system to the distributed object storage, and then the target data fragments can be pulled from the distributed object storage according to the configured read mode. This can shorten the distance for data use and thus effectively improve the utilization rate of the computing resources of the target distributed computing framework.
[0092] Please refer to Figure 12 This is a schematic diagram of the overall architecture of the data processing method according to an embodiment of this application. As shown in the figure, it covers the entire chain from data input, processing, calculation, training to management: The left side is the multi-source data input layer, which includes multimodal data such as video, images, and audio, as well as structured or semi-structured data such as embedded vectors and metadata. The middle is the core functional layer, which, from bottom to top, consists of the "distributed computing framework" supporting resources (responsible for resource scheduling and distributed environment management), the "column storage system" of the storage layer (providing high-efficiency IO, index acceleration, and other capabilities), the "multimodal computing" of the computing layer (integrating data dictionaries, data, and services), the "data processing pipeline" of the processing layer (implementing operator configuration, stream / batch processing, etc.), and the "training module" of the training layer (responsible for data loading and reinforcement learning training). The right side is the supporting management module, covering the management and operation of the entire process, including operators, pipelines, and features.
[0093] The overall architecture is centered on "multimodal data". Through the synergy of distributed computing, storage and processing capabilities, and with the support of a complete management module, it achieves integrated support from multimodal data access to training applications.
[0094] Example 2 Figure 13 The diagram schematically illustrates a data processing apparatus according to Embodiment 2 of this application. This apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program module referred to in the embodiments of this application refers to a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the functions of each program module in this embodiment. For example... Figure 13 As shown, the device 900 may include an update module 910 and a storage module 920, wherein: The update module 910 is used to update the multimodal data to the target column when the multimodal data generated by the target distributed computing framework is processed and the multimodal data already corresponds to the target column of the columnar storage system. The columnar storage system supports the storage of heterogeneous data, and the target distributed computing framework supports streaming processing and batch processing. Storage module 920 is used to store the multimodal data into a newly added column of the columnar storage system when the multimodal data does not correspond to any column of the columnar storage system; The processing module 930 is used to, when it is necessary to acquire target multimodal data, read the target multimodal data from the columnar storage system using a target distributed computing framework, process the target multimodal data, and write the processed result back into the columnar storage system.
[0095] In an optional embodiment, the processing module 930 is further configured to: In cases where it is necessary to obtain target multimodal data, the target distributed computing framework is used to read target data fragments from the columnar storage system. The target data fragment is one of the fragments of the target multimodal data, and the target data fragment includes a target fragment ID. The target distributed computing framework is used to perform target computing operations on the target data shards according to the target shard ID to obtain the operation results. The operation results are merged based on the target shard ID and then written into the columnar storage system.
[0096] In an optional embodiment, the device 900 is further used for: The first data of the target data fragments is converted from column format to second data of dictionary data structure using the target distributed computing framework. The target distributed computing framework is used to perform target computation operations on the second data based on the target shard ID to obtain the operation result. The target distributed computing framework is used to convert the operation results back into third data in column format; The third data is merged based on the target shard ID and then written into the columnar storage system.
[0097] In an optional embodiment, the device 900 is further used for: The target distributed computing framework is used to invoke a pre-configured data processing pipeline, wherein the data processing pipeline is configured using a data serialization language. The data processing pipeline performs target calculation operations on the second data based on the target shard ID.
[0098] In an optional embodiment, the data processing pipeline includes standardized operators, which include general operators and custom operators. The general operators are operators pre-built in the target distributed computing framework, and the custom operators are operators configured using the operator standardization interface in the target distributed computing framework. The configuration of the operator standardization interface includes operator initialization configuration, operator input, operator processing logic, and operator output. The operator processing logic includes single-item processing and batch processing. Correspondingly, the device 900 is also used for: Obtain the configuration of the standardized operator; The standardized operator is invoked based on its configuration. The standardized operator performs target calculation operations on the second data based on the target fragment ID.
[0099] In an optional embodiment, the configuration of the data processing pipeline includes execution engine type, engine execution mode, data processing mode, data input configuration, operator concatenation configuration, and data output configuration. The engine execution mode includes cluster mode and local mode, and the data processing mode includes batch processing mode and stream processing mode. Correspondingly, the device 900 is also used for: Obtain the configuration of the data pipeline; The target distributed computing framework invokes the data pipeline based on the configuration of the data pipeline.
[0100] In an optional embodiment, when the data processing mode is the streaming processing mode, the configuration of the data processing pipeline further includes breakpoint resume, elastic scaling, queue capacity control, and block size control.
[0101] In an optional embodiment, the device 900 is further used for: The target data fragments are pre-read from the columnar storage system using a data loader; The target data is sharded and cached in the distributed object storage of the target distributed computing framework; The data loader is used to pull the target data fragment from the distributed object storage according to a preset data reading mode and populate it into the local cache queue; The target data fragment is obtained from the cache queue using the target distributed computing framework.
[0102] In an optional embodiment, the configuration of the data loader includes the read path from the columnar storage system, the field columns to be read, the data filtering conditions, the total number of samples to be read, the distributed training configuration, the batch data to be read beforehand, the data reading mode, the data transformation, and the hardware adaptation. The data reading mode includes asynchronous reading and streaming reading. The device 900 is also used for: Obtain the configuration of the data loader, and call the data loader to pre-read target data fragments from the columnar storage system according to the configuration of the data loader; The data reading mode is determined according to the configuration of the data loader, and the target data fragment is pulled from the distributed object storage according to the configured data reading mode.
[0103] Example 3 Figure 14 This illustration schematically depicts the hardware architecture of a computer device 10000 suitable for implementing a data processing method according to Embodiment 3 of this application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, or robot. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. Figure 14 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein: The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for data processing methods. Furthermore, the memory 10010 can also be used to temporarily store various types of data that have already been output or will be output.
[0104] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.
[0105] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.
[0106] It should be pointed out that, Figure 14 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.
[0107] In this embodiment, the data processing method stored in memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of this application.
[0108] Example 4 This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the data processing method in the embodiments.
[0109] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the data processing method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.
[0110] Example 5 This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.
[0111] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.
[0112] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.
Claims
1. A data processing method, characterized in that, The method includes: In the case where multimodal data is generated by processing using a target distributed computing framework and the multimodal data already corresponds to a target column of a columnar storage system, the multimodal data is updated to the target column. The columnar storage system supports the storage of heterogeneous data, and the target distributed computing framework supports streaming processing and batch processing. If the multimodal data does not correspond to any column of the columnar storage system, the multimodal data is stored in a newly added column of the columnar storage system. In cases where target multimodal data needs to be acquired, the target distributed computing framework is used to read the target multimodal data from the columnar storage system, the target multimodal data is processed, and the result of the processing is written into the columnar storage system.
2. The method according to claim 1, characterized in that, In the case where target multimodal data needs to be acquired, the method of reading the target multimodal data from the columnar storage system using a target distributed computing framework, processing the target multimodal data, and writing the processed result into the columnar storage system includes: In cases where it is necessary to obtain target multimodal data, the target distributed computing framework is used to read target data fragments from the columnar storage system. The target data fragment is one of the fragments of the target multimodal data, and the target data fragment includes a target fragment ID. The target distributed computing framework is used to perform target computing operations on the target data shards according to the target shard ID to obtain the operation results. The operation results are merged based on the target shard ID and then written into the columnar storage system.
3. The method according to claim 2, characterized in that, The step of using the target distributed computing framework to perform target computing operations on the target data shards according to the target shard ID, and obtaining the operation result, includes: The first data of the target data fragments is converted from column format to second data of dictionary data structure using the target distributed computing framework. The target distributed computing framework is used to perform target computation operations on the second data based on the target shard ID to obtain the operation result. Correspondingly, the step of merging the operation results based on the target shard ID and writing them into the columnar storage system includes: The target distributed computing framework is used to convert the operation results back into third data in column format; The third data is merged based on the target shard ID and then written into the columnar storage system.
4. The method according to claim 3, characterized in that, The step of using the target distributed computing framework to perform target computation operations on the second data based on the target shard ID includes: The target distributed computing framework is used to invoke a pre-configured data processing pipeline, wherein the data processing pipeline is configured using a data serialization language. The data processing pipeline performs target calculation operations on the second data based on the target shard ID.
5. The method according to claim 4, characterized in that, The data processing pipeline includes standardized operators, which include general operators and custom operators. The general operators are pre-built operators in the target distributed computing framework, and the custom operators are operators configured using the operator standardization interface in the target distributed computing framework. The configuration of the operator standardization interface includes operator initialization configuration, operator input, operator processing logic, and operator output. The operator processing logic includes single-item processing and batch processing. Correspondingly, the step of utilizing the target distributed computing framework to invoke a pre-configured data processing pipeline, and performing target computation operations on the second data according to the target shard ID through the data processing pipeline, includes: Obtain the configuration of the standardized operator; The standardized operator is invoked based on its configuration. The standardized operator performs target calculation operations on the second data based on the target fragment ID.
6. The method according to claim 4, characterized in that, The configuration of the data processing pipeline includes execution engine type, engine execution mode, data processing mode, data input configuration, operator concatenation configuration, and data output configuration. The engine execution mode includes cluster mode and local mode, and the data processing mode includes batch processing mode and stream processing mode. Correspondingly, the step of using the target distributed computing framework to invoke a pre-configured data processing pipeline includes: Obtain the configuration of the data pipeline; The target distributed computing framework invokes the data pipeline based on the configuration of the data pipeline.
7. The method according to claim 6, characterized in that, When the data processing mode is the stream processing mode, the configuration of the data processing pipeline also includes breakpoint resume, elastic scaling, queue capacity control, and block size control.
8. The method according to claim 2, characterized in that, The step of reading target data fragments from the columnar storage system using the target distributed computing framework includes: The target data fragments are pre-read from the columnar storage system using a data loader; The target data is sharded and cached in the distributed object storage of the target distributed computing framework; The data loader is used to pull the target data fragment from the distributed object storage according to a preset data reading mode and populate it into the local cache queue; The target data fragment is obtained from the cache queue using the target distributed computing framework.
9. The method according to claim 8, characterized in that, The configuration of the data loader includes the read path from the columnar storage system, the field columns to be read, the data filtering conditions, the total number of samples to be read, the distributed training configuration, the batch data to be read before reading, the data reading mode, the data transformation, and the hardware adaptation. The data reading mode includes asynchronous reading and streaming reading. Correspondingly, the step of pre-reading target data fragments from the columnar storage system using a data loader includes: Obtain the configuration of the data loader, and call the data loader to pre-read target data fragments from the columnar storage system according to the configuration of the data loader; The step of using the data loader to pull the target data shard from the distributed object storage according to a preset data reading mode includes: The data reading mode is determined according to the configuration of the data loader, and the target data fragment is pulled from the distributed object storage according to the configured data reading mode.
10. A data processing apparatus, characterized in that, The device includes: An update module is used to update the multimodal data to the target column when the multimodal data generated by the target distributed computing framework is processed and the multimodal data already corresponds to the target column of the columnar storage system. The columnar storage system supports the storage of heterogeneous data, and the target distributed computing framework supports streaming processing and batch processing. A storage module is used to store the multimodal data into a newly added column of the columnar storage system when the multimodal data does not correspond to any column of the columnar storage system. The processing module is used to read the target multimodal data from the columnar storage system using the target distributed computing framework when it is necessary to obtain the target multimodal data, process the target multimodal data, and write the processed result of the target multimodal data into the columnar storage system.
11. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 9.
13. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9.