Data processing method and device and computer readable storage medium
By using a configuration-driven data processing method, the development difficulty caused by logical coupling in the traditional real-time data processing system is solved, enabling flexible loading, dynamic transformation, and diversified writing, thereby improving development efficiency and business response speed.
Patent Information
- Application Number
- CN202510762721.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-09
- Publication Date
- 2025-10-31
AI Technical Summary
In traditional real-time data processing systems, the logic of various data sources and target terminals is highly coupled, which increases development difficulty and limits the speed of business iteration, making it difficult to respond quickly to new business needs.
By adopting a configuration-based data processing method, and by obtaining business configurations, we can achieve flexible loading of source data, dynamic transformation of intermediate logic, and diversified writing to the target end, thereby improving development efficiency.
It achieves a high degree of modularity and scalability in the data processing process, reduces development difficulty, improves development efficiency and business response speed, and supports diverse real-time data processing needs.
Smart Images
Figure CN120873053A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus and computer-readable storage medium. Background Technology
[0002] In traditional real-time data processing systems, various data sources and target endpoints are typically implemented through hard-coded connectors and transformation logic, which are scattered across different modules or jobs. For example, if support for a new message queue type or database is required, a dedicated connector must be written or extended, and corresponding field mappings and filtering rules must be manually added to the transformation logic. Due to the high degree of coupling between the source, transformation layer, and target endpoint logic, each time a new business is introduced or an existing process is modified, large-scale changes to the core code and repackaging and redeployment are required.
[0003] This deeply coupled development approach directly leads to a significant increase in development difficulty. On the one hand, developers not only need to be familiar with the business logic implementation details of the entire project, but also need to modify code back and forth between multiple modules, which can easily introduce low-level errors. On the other hand, in order to ensure that newly added code does not affect existing functions, comprehensive regression testing must also be carried out, further extending the development cycle.
[0004] Therefore, in existing technologies, the speed of business iteration is often forced to be limited, making it difficult for teams to respond quickly to new business needs. Summary of the Invention
[0005] This application provides a data processing method, apparatus, and computer-readable storage medium that can improve development efficiency.
[0006] The first aspect of this application provides a data processing method, including: Obtain the service configuration, which includes source configuration, intermediate conversion logic configuration, and target configuration; Load the data to be processed according to the source configuration; The data to be processed is processed according to the intermediate transformation logic configuration to obtain the processed data; The processed data is written according to the target configuration.
[0007] Optionally, loading the data to be processed according to the source configuration includes: The data source and data loading strategy are determined based on the source configuration. Data to be processed is loaded from the data source according to the data loading strategy.
[0008] Optionally, loading the data to be processed from the data source according to the data loading strategy includes: Obtain raw data from the data source; The original data is filtered according to the first filtering strategy in the data loading strategy in order to load the data to be processed.
[0009] Optionally, the step of processing the data to be processed according to the intermediate transformation logic configuration to obtain processed data includes: The data to be processed is processed according to the preprocessing strategy in the intermediate transformation logic configuration to obtain preprocessed data; The preprocessed data is processed according to the business logic interface in the intermediate conversion logic configuration to obtain the processed data.
[0010] Optionally, the step of processing the data to be processed according to the preprocessing strategy in the intermediate transformation logic configuration to obtain preprocessed data includes: The format conversion strategy and the second filtering strategy are determined based on the preprocessing strategy in the intermediate conversion logic configuration. According to the format conversion strategy, the format converter is invoked to perform format conversion on the data to be processed, and the converted data is obtained. According to the second filtering strategy, the filtering rules are invoked to perform illegal data filtering on the transformed data in order to obtain preprocessed data.
[0011] Optionally, the step of writing the processed data according to the target terminal configuration includes: The target source and corresponding writing strategy of the processed data are determined according to the target end configuration, wherein the target source includes one or more, and the target source corresponds one-to-one with the writing strategy; The processed data is written to the target source according to the writing strategy.
[0012] Optionally, writing the processed data to the target source according to the writing strategy includes: The fields to be written are extracted from the processed data according to the writing strategy; Write the field to be written to the target source.
[0013] A second aspect of this application provides a data processing apparatus, comprising: The acquisition unit is used to acquire service configuration, which includes source configuration, intermediate conversion logic configuration and target configuration. The loading unit is used to load the data to be processed according to the source configuration; The processing unit is configured to perform processing on the data to be processed according to the intermediate conversion logic configuration, so as to obtain the processed data; The writing unit is used to write the processed data according to the target terminal configuration.
[0014] Optionally, the loading unit includes: The first determining module is used to determine the data source and data loading strategy based on the source configuration. A loading module is used to load data to be processed from the data source according to the data loading strategy.
[0015] Optionally, the loading module is specifically used for: Obtain raw data from the data source; The original data is filtered according to the first filtering strategy in the data loading strategy in order to load the data to be processed.
[0016] Optionally, the processing unit includes: The first processing module is used to process the data to be processed according to the preprocessing strategy in the intermediate conversion logic configuration to obtain preprocessed data. The second processing module is used to process the preprocessed data according to the business logic interface in the intermediate conversion logic configuration to obtain the processed data.
[0017] Optionally, the first processing module is specifically used for: The format conversion strategy and the second filtering strategy are determined based on the preprocessing strategy in the intermediate conversion logic configuration. According to the format conversion strategy, the format converter is invoked to perform format conversion on the data to be processed, and the converted data is obtained. According to the second filtering strategy, the filtering rules are invoked to perform illegal data filtering on the transformed data in order to obtain preprocessed data.
[0018] Optionally, the writing unit includes: The second determining module is used to determine the target source and corresponding writing strategy of the processed data according to the target end configuration, wherein the target source includes one or more, and the target source corresponds one-to-one with the writing strategy; The writing module is used to write the processed data to the target source according to the writing strategy.
[0019] Optionally, the writing module is specifically used for: The fields to be written are extracted from the processed data according to the writing strategy; Write the field to be written to the target source.
[0020] A third aspect of this application provides a data processing apparatus, comprising: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method of the first aspect and any possible implementation of the first aspect.
[0021] A fourth aspect of this application provides a computer-readable storage medium storing a program that, when executed on a computer, causes the computer to perform the methods of the first aspect and any possible implementation thereof.
[0022] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: This application utilizes configuration-driven processing to achieve flexible loading of source data, dynamic transformation of intermediate logic, and diverse writing to the target end, making the data processing process highly modular and scalable. This configuration-based design allows business developers to complete complex data flow and processing simply by adjusting the configuration, greatly reducing development difficulty and thus improving development efficiency. Attached Figure Description
[0023] Figure 1 This is a flowchart illustrating one embodiment of the data processing method in this application; Figure 2 This is a flowchart illustrating one embodiment of loading data to be processed in this application; Figure 3 This is a flowchart illustrating one embodiment of obtaining processed data in this application; Figure 4 This is a schematic flowchart of one embodiment of obtaining preprocessed data in this application; Figure 5 This is a flowchart illustrating an embodiment of writing processed data in this application; Figure 6 This is a schematic diagram of the structure of one embodiment of the data processing apparatus in this application; Figure 7 This is a schematic diagram of another embodiment of the data processing apparatus in this application. Detailed Implementation
[0024] This application provides a data processing method, apparatus, and computer-readable storage medium to improve development efficiency.
[0025] The method described in this application can be applied to servers, terminals, or other devices with logical processing capabilities; therefore, this application does not limit its application. For ease of description, the following description uses a business server as the executing entity.
[0026] The embodiments of this application will now be described with reference to the accompanying drawings.
[0027] Please see Figure 1 , Figure 1 One embodiment of the data processing method provided in this application includes: 101. Obtain business configuration, which includes source configuration, intermediate conversion logic configuration, and target configuration; The business server reads complete business configuration information from configuration storage (such as a database or configuration service) by calling the configuration center interface. The source configuration includes information such as the source connection address, authentication credentials, and data format; the intermediate transformation logic configuration includes information such as field mapping rules, filtering conditions, and verification rules; and the target configuration includes information such as the target storage type, connection parameters, and write mode (such as overwrite or append), providing necessary parameters for subsequent data processing.
[0028] 102. Load the data to be processed according to the source configuration; Based on the obtained source configuration, the business server establishes a connection with the source data source (such as a relational database, NoSQL storage, file system, or message queue), uses the corresponding authentication information and access address to perform data extraction operations, reads the qualified raw data into the server memory or cache, and performs preliminary data verification (such as format verification and field integrity verification) to ensure that the data to be processed meets the expected requirements.
[0029] 103. Configure the data to be processed according to the intermediate transformation logic to obtain the processed data; Based on the intermediate transformation logic configuration, the business server traverses or batch processes the raw data loaded into memory or cache. For each record of raw data, it performs application field mapping (such as mapping "name" to "user_name"), data cleaning (such as removing null values and formatting timestamps), data filtering (such as removing records that do not meet the conditions), and necessary business validation (such as uniqueness validation and range validation). It also performs calculations according to aggregation or derivative requirements (such as cumulative statistics and generating derived fields). Finally, it summarizes the processed data that conforms to the target format into a dataset to be written.
[0030] 104. Write the processed data according to the target configuration.
[0031] The business server serializes the processed data according to the target format (such as JSON, CSV, relational table structure) based on the target type (such as target database, data warehouse, file storage, message queue, etc.) and corresponding connection information and write mode (overwrite / append / partition write, etc.) in the target configuration, and writes the data to the target through the database driver or storage interface. During the writing process, the business server will execute batch commit, transaction control or idempotent write strategies according to the configuration to ensure data consistency and reliability.
[0032] In this embodiment, the business server uses configuration-driven mechanisms to flexibly load source data, dynamically transform intermediate logic, and perform diverse writes to the target end, making the data processing process highly modular and scalable. This configuration-based design allows business developers to complete complex data flows and processing simply by adjusting the configuration, greatly reducing development difficulty and thus improving development efficiency. Furthermore, it supports dynamic configuration updates to meet diverse real-time data processing needs.
[0033] Please see Figure 2 In some embodiments of this application, step 102 in the above embodiments, which loads the data to be processed according to the source configuration, may include the following steps: 201. Determine the data source and data loading strategy based on the source configuration; The business server parses the source configuration to identify the data source (such as different clusters, different types of databases, file systems, or message queues) and its corresponding data loading strategy. The system combines factory and strategy patterns to dynamically select and create corresponding data loading strategy instances based on the configuration, enabling unified management and flexible invocation of multiple data sources. This provides accurate connection parameters and access methods for subsequent data acquisition, facilitating future expansion by simply adding corresponding strategies and implementations without modifying the core code.
[0034] 202. Obtain raw data from the data source; Based on a defined data source and corresponding strategy, the business server establishes a connection and performs data reading operations, retrieves raw data from a specified cluster and type of data source, and pulls the data into the server's memory or cache to prepare for subsequent processing.
[0035] 203. Filter the original data according to the first filtering strategy in the data loading strategy to load the data to be processed.
[0036] The business server filters the raw data it pulls according to the filtering rules defined in the configuration, removes data that does not meet the conditions, and retains only the valid data that meets the business requirements, so as to ensure the accuracy of data and the efficiency of processing in subsequent processing stages.
[0037] In this embodiment, through explicit data source configuration and loading strategies, the business server can flexibly obtain the required data from various types of data sources, and ensure the accuracy and effectiveness of the loaded data through filtering strategies, thereby improving the precision and efficiency of data processing and reducing the difficulty of business development and maintenance.
[0038] Please see Figure 3 In some embodiments of this application, step 103 in the above embodiments, which configures the data to be processed according to the intermediate conversion logic to obtain processed data, may include the following steps: 301. Process the data to be processed according to the preprocessing strategy in the intermediate transformation logic configuration to obtain preprocessed data; The business server parses the preprocessing strategy in the intermediate transformation logic configuration and performs preliminary processing on the loaded data to be processed, including removing invalid or abnormal data, unifying the format conversion, field validation and basic calculations, to ensure that the data structure and content meet the requirements of subsequent business processing, improve data quality and consistency, and provide a standardized data foundation for subsequent complex business logic processing.
[0039] 302. Process the preprocessed data according to the business logic interface in the intermediate conversion logic configuration to obtain the processed data.
[0040] Based on the configuration of the business logic interface, the business server performs specific business rule processing on the preprocessed data, including various operations such as field mapping, condition filtering, data association, real-time statistical calculation, function calls and asynchronous queries, and finally generates processed data that meets business requirements, supporting flexible business scenarios and dynamic adjustments.
[0041] Specifically, the business server can first read the business code in the configuration, then find the corresponding component instance in a set of "business logic components" maintained internally, and call its processing method to process each piece of preprocessed data.
[0042] For example, if the configuration defines an "order merging" business code, the business server will load a component specifically responsible for integrating order information with related shipping and inventory information. When an order record arrives, this component will retrieve and merge relevant information from the associated data stream based on a pre-agreed association key, ultimately outputting a complete order object containing all details. As another example, for the "user behavior statistics" scenario, the configuration will point to a component for real-time aggregation of user behavior. This component will accumulate or group each message according to the statistical dimensions specified in the configuration, outputting the statistical results.
[0043] If a piece of preprocessed data needs to be supplemented with external dimension information (such as adding extra attributes for each user), and the configuration includes "asynchronous dimension query" business code, the business server will call a component that specifically handles asynchronous queries. This component will request dimension data from external storage in parallel, populate the corresponding record after receiving the results, and then output a complete data object containing dimension information.
[0044] All these components adhere to a unified interface specification. Once registered internally and assigned corresponding business code, they can be dynamically loaded by the business server at runtime. When new business scenarios emerge, such as "log content encryption" or "custom function calculation," simply adding the new business logic component to the component list and configuring its corresponding code allows the business server to automatically invoke it in the processing chain according to the configuration, without modifying the core process. This design achieves both isolation between different business logics and allows multiple business logics to be deployed within the same project, thereby saving resources and facilitating management. Furthermore, each component only needs to focus on its own business logic, and components interact through a unified specification, making the overall processing flow both clear and easily extensible.
[0045] In this embodiment, through phased preprocessing and business logic processing, the business server can effectively improve data quality and processing standardization. It utilizes configuration-driven processing to achieve flexible and diverse processing logic, reducing the dependence of business development on code, enhancing the scalability and maintainability of the system, while ensuring the accuracy and business applicability of the processed data, and improving the overall data processing efficiency and reliability.
[0046] Please see Figure 4 In some embodiments of this application, step 301 in the above embodiments, which processes the data to be processed according to the preprocessing strategy in the intermediate conversion logic configuration to obtain preprocessed data, may include the following steps: 401. Determine the format conversion strategy and the second filtering strategy based on the preprocessing strategy in the intermediate conversion logic configuration; The business server parses the preprocessing strategy configuration to identify the format converters and illegal data filtering rules that need to be invoked. For example, in the implementation class of KafkaDeserializationSchema, the business server can directly discard Kafka messages unrelated to the project upstream according to the configuration, thereby reducing the pressure on subsequent processing. At the same time, the business server dynamically selects the corresponding format converter by invoking the factory + strategy mechanism based on the message type (such as Maxwell, OGG, OMS, etc.), ensuring that messages from different sources can be correctly identified and enter the appropriate conversion process. For filtering rules, the business server extracts the fields and thresholds that need to be checked, laying the foundation for subsequent illegal data removal.
[0047] 402. According to the format conversion strategy, call the format converter to perform format conversion on the data to be processed, and obtain the converted data; The business server performs formatting operations on the raw data based on the selected transformer. For example, when consuming multi-topic messages from Kafka, the business server uses a custom partitioning strategy instead of Flink's default partitioning algorithm to avoid data skew and improve processing throughput. During the specific transformation process, the business server uses SpringEL expressions to execute configurable functions, such as dynamically generating RowKeys or encrypting / decrypting sensitive fields, allowing business logic to change flexibly with configuration. After the transformation is complete, the business server outputs the results in a unified standard data format, providing consistent input for subsequent filtering and processing.
[0048] 403. According to the second filtering strategy, the filtering rules are invoked to perform illegal data filtering on the transformed data in order to obtain preprocessed data.
[0049] The application server checks each transformed data entry according to the filtering rules defined in the configuration. For example, the application server uses Flink's side output stream to send invalid records that do not meet the conditions to the exception queue, while keeping valid data in the main stream. At the same time, the application server starts FlinkCDC to listen to the MySQL Binlog. When the configuration table changes, the latest configuration is synchronized to each TaskManager through a broadcast stream, so that the latest rules can be applied to each message in real time during processing. The preprocessing logic is organized using the chain of responsibility pattern. If a new preprocessing node needs to be added, the application server only needs to append the corresponding node to the chain without modifying the existing logic, thus achieving flexible expansion.
[0050] In this embodiment, the business server first determines the format conversion and filtering strategies based on the preprocessing strategy, and then performs format conversion and illegal data filtering on the data. This ensures that the input data is uniformly formatted and strictly filtered before entering subsequent processing, thereby improving data quality and processing accuracy, and providing a basis for flexibly expanding the preprocessing logic.
[0051] Please see Figure 5 In some embodiments of this application, step 104 in the above embodiments, which involves writing the processed data according to the target end configuration, may include the following steps: 501. Determine the target source and corresponding writing strategy of the processed data according to the target end configuration. The target source includes one or more, and the target source corresponds one-to-one with the writing strategy. The business server parses the configuration list, identifies one or more target sources, and associates a corresponding write strategy with each target source. For example, when the configuration includes multiple clusters or different data types, the processed data can be split into their respective targets, and the write method and parallelism can be specified for each target to ensure that the performance requirements of different write scenarios are met.
[0052] 502. Extract the fields to be written from the processed data according to the writing strategy; Based on the write strategy determined in the previous step, the business server extracts and maps fields for each piece of processed data. For example, if a target only needs some fields, the user identifier and timestamp can be filtered from the complete record according to the strategy; if another target needs to add business tags and regional information, the corresponding fields can be extracted at this stage to ensure that each target receives a subset of data that meets its needs.
[0053] 503. Write the fields to be written to the target source.
[0054] The business server writes the fields corresponding to each target in parallel according to the configuration. For example, parallel writing to different databases or storage systems can be achieved through a custom write component, and batch commit or transaction control strategies can be adopted to balance throughput and consistency. When it is necessary to expand to new target sources, only the corresponding strategy implementation needs to be added, without intruding on the existing write logic.
[0055] In this embodiment, the server first determines the multi-target write strategy, then accurately extracts the fields to be written, and finally writes them to each target source in parallel. This can achieve high concurrency and multiplexing writes under different clusters and data types, thereby significantly improving data distribution efficiency and system scalability.
[0056] Please see Figure 6 , Figure 6 One embodiment of the data processing apparatus provided in this application includes: The acquisition unit 601 is used to acquire business configuration, which includes source configuration, intermediate conversion logic configuration and target configuration. Loading unit 602 is used to load data to be processed according to the source configuration; Processing unit 603 is configured to perform processing on the data to be processed according to intermediate conversion logic to obtain processed data; The writing unit 604 is used to write the processed data according to the target configuration.
[0057] In this embodiment, the business server uses configuration-driven mechanisms to flexibly load source data, dynamically transform intermediate logic, and perform diverse writes to the target end, making the data processing process highly modular and scalable. This configuration-based design allows business developers to complete complex data flows and processing simply by adjusting the configuration, greatly reducing development difficulty and thus improving efficiency. Furthermore, it supports dynamic configuration updates to meet diverse real-time data processing needs.
[0058] Optionally, loading unit 602 includes: The first determination module is used to determine the data source and data loading strategy based on the source configuration. The loading module is used to load data to be processed from the data source according to the data loading strategy.
[0059] Optionally, the loaded module is specifically used for: Retrieve raw data from the data source; The original data is filtered according to the first filtering strategy in the data loading strategy in order to load the data to be processed.
[0060] Optionally, the processing unit 603 includes: The first processing module is used to process the data to be processed according to the preprocessing strategy in the intermediate transformation logic configuration in order to obtain preprocessed data. The second processing module is used to process the preprocessed data according to the business logic interface in the intermediate conversion logic configuration in order to obtain the processed data.
[0061] Optionally, the first processing module is specifically used for: The format conversion strategy and the second filtering strategy are determined based on the preprocessing strategy in the intermediate conversion logic configuration. Based on the format conversion strategy, the format converter is invoked to perform format conversion on the data to be processed, and the converted data is obtained. The second filtering strategy calls the filtering rules to perform illegal data filtering on the transformed data in order to obtain preprocessed data.
[0062] Optionally, the write unit 604 includes: The second determining module is used to determine the target source and corresponding writing strategy of the processed data according to the target end configuration. The target source includes one or more, and the target source corresponds one-to-one with the writing strategy. The write module is used to write the processed data to the target source according to the write strategy.
[0063] Optionally, the write module is specifically used for: Extract the fields to be written from the processed data according to the writing strategy; Write the fields to be written to the target source.
[0064] In this implementation, the functions of each unit and module are as described above. Figures 1 to 5 The steps in the illustrated embodiments are the same and will not be repeated here.
[0065] Please see Figure 7 , Figure 7 Another embodiment of the data processing apparatus provided in this application includes: Processor 701, memory 702, input / output unit 703, and bus 704; The processor 701 is connected to the memory 702, the input / output unit 703 and the bus 704; The memory 702 stores a program, and the processor 701 calls the program to execute it. Figures 1 to 5 The steps in the illustrated embodiment.
[0066] In this embodiment, the function of processor 701 is the same as described above. Figures 1 to 5 The steps in the illustrated embodiments are the same and will not be repeated here.
[0067] This application also provides a computer-readable storage medium on which a program is stored. When the program is executed on a computer, it causes the computer to perform the aforementioned actions. Figures 1 to 5 The method in any possible implementation.
[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0069] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.
[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0071] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A data processing method, characterized in that, include: Obtain the service configuration, which includes source configuration, intermediate conversion logic configuration, and target configuration; Load the data to be processed according to the source configuration; The data to be processed is processed according to the intermediate transformation logic configuration to obtain the processed data; The processed data is written according to the target configuration.
2. The method according to claim 1, characterized in that, The step of loading the data to be processed according to the source configuration includes: The data source and data loading strategy are determined based on the source configuration. Data to be processed is loaded from the data source according to the data loading strategy.
3. The method according to claim 2, characterized in that, The step of loading the data to be processed from the data source according to the data loading strategy includes: Obtain raw data from the data source; The original data is filtered according to the first filtering strategy in the data loading strategy in order to load the data to be processed.
4. The method according to claim 1, characterized in that, The step of processing the data to be processed according to the intermediate transformation logic configuration to obtain processed data includes: The data to be processed is processed according to the preprocessing strategy in the intermediate transformation logic configuration to obtain preprocessed data; The preprocessed data is processed according to the business logic interface in the intermediate conversion logic configuration to obtain the processed data.
5. The method according to claim 4, characterized in that, The step of processing the data to be processed according to the preprocessing strategy in the intermediate transformation logic configuration to obtain preprocessed data includes: The format conversion strategy and the second filtering strategy are determined based on the preprocessing strategy in the intermediate conversion logic configuration. According to the format conversion strategy, the format converter is invoked to perform format conversion on the data to be processed, and the converted data is obtained. According to the second filtering strategy, the filtering rules are invoked to perform illegal data filtering on the transformed data in order to obtain preprocessed data.
6. The method according to any one of claims 1 to 5, characterized in that, The step of writing the processed data according to the target terminal configuration includes: The target source and corresponding writing strategy of the processed data are determined according to the target end configuration, wherein the target source includes one or more, and the target source corresponds one-to-one with the writing strategy; The processed data is written to the target source according to the writing strategy.
7. The method according to claim 6, characterized in that, The step of writing the processed data to the target source according to the writing strategy includes: The fields to be written are extracted from the processed data according to the writing strategy; Write the field to be written to the target source.
8. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire service configuration, which includes source configuration, intermediate conversion logic configuration and target configuration. The loading unit is used to load the data to be processed according to the source configuration; The processing unit is configured to perform processing on the data to be processed according to the intermediate conversion logic configuration, so as to obtain the processed data; The writing unit is used to write the processed data according to the target terminal configuration.
9. A data processing apparatus, characterized in that, include: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1 to 7.