Cross-platform self-adaptive data processing working system and method
By encapsulating the interfaces and converting the protocols of the computing platform, the automation and intelligence of cross-platform data processing are achieved, solving the problems of high labor costs and performance degradation in cross-platform collaborative work, and improving data processing efficiency and stability.
Patent Information
- Application Number
- CN202511716131.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-17
AI Technical Summary
In existing technologies, cross-platform data processing tasks require professionals to manually write scripts and debug algorithms, resulting in high labor costs and performance degradation due to platform limitations. Furthermore, it is difficult to conduct efficient collaborative work between different platforms.
By parsing and encapsulating the native interfaces of each computing platform, standardized interfaces are generated, communication protocols are converted, data transmission and format conversion are automated, parameter adjustment schemes are generated according to platform characteristics and algorithm requirements, task information is input using a graphical interface, and structured task description files are generated.
It reduces the coding burden on users in cross-platform collaborative work, improves data processing efficiency and stability, shortens task configuration time, avoids performance bottlenecks caused by platform limitations, and has good scalability and adaptability.
Smart Images
Figure CN121547522A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer system technology, and more specifically, relates to a cross-platform adaptive data processing system and method. Background Technology
[0002] With the development of internet information technology, people's production and daily life have generated more and more complex data. In order to use this data more efficiently and to extract the value from it, people have studied a large number of algorithms such as data cleaning, data correction, data standardization, and data imputation.
[0003] However, there is currently no single platform capable of handling all data processing tasks. Professionals often need to work across platforms in stages when processing data. Current data processing tasks are no longer limited to using a single computing platform, but are completed collaboratively using multiple computing platforms.
[0004] For professionals, building a cross-platform computing workflow that uses multiple algorithms for data processing and coordinating the various computational components to achieve high performance is challenging. Different platforms require specific code implementations, necessitating users to master multiple programming languages and be familiar with various platforms, thus increasing labor costs. Users also need to write additional scripts to enable collaboration between different platforms, such as data transfer and data format conversion, which adds an extra burden to professionals. Furthermore, users need to debug the algorithm's computing platform to achieve ideal performance and results. While some performance degradation is caused by user code, other factors are platform limitations, which are difficult for users to debug. Summary of the Invention
[0005] To address the aforementioned problems and technical deficiencies, this application adopts the following technical solution: a cross-platform adaptive data processing method, comprising the following steps: The native interfaces of each computing platform are parsed, the functional parameter information of the interfaces is extracted, and a standardized encapsulated interface is generated. Based on the requirements of data processing tasks, the communication protocols of different platforms are converted into a unified communication protocol within the system to establish a data transmission channel; The data format is converted based on the format conversion rules, and after the conversion is completed, it is transmitted to the target platform through the established data transmission channel; Based on the characteristics of the target platform and the algorithm requirements, a parameter adjustment plan is generated and sent to the corresponding computing platform. The computing platform then configures the parameters according to the parameter adjustment plan. The system inputs data information for data processing tasks through a graphical interface, parses the input information, and generates a structured task description file.
[0006] Preferably, the generation of the standardized interface encapsulation includes: Version tracing is performed on the parsed native interface function parameter information to extract the interface release time, update logs, and version iteration differences, and an interface version mapping table is constructed. The differences in functional parameters between different versions are matched a second time with the preset standardized interface template, and the functions of the native interface are mapped to the corresponding modules of the standardized interface to generate the encapsulated standardized interface.
[0007] Furthermore, after the standardized interface is encapsulated and generated, the interface access permission configuration of each computing platform is parsed, a permission mapping matrix is established, the platform's native permission rules are converted into a unified system permission identification system, the permission level of the request initiator is automatically verified when the interface is called, a standardized permission deficiency response is returned for call requests that exceed the permission scope, and permission verification logs are recorded.
[0008] Preferably, the communication protocol conversion includes: It monitors communication requests between various computing platforms and the system in real time. When it receives a request for a data processing task, it parses the communication protocol type carried in the request and converts the communication protocols of different platforms into a unified communication protocol within the system according to the mapping relationship in the preset protocol conversion rules. When there is a many-to-one mapping conflict in the preset protocol conversion rules, the contextual semantics of the parsed fields are analyzed, and the optimal mapping rule is selected by combining the success rate statistics of historical conversions.
[0009] Preferably, the format conversion rules are pre-stored in the rule base, which contains information on the data formats supported by each computing platform and the conversion rules between different formats. At the same time, historical cases of format conversion are collected, and the format combinations that are not covered are identified through cluster analysis. When a new format conversion requirement is obtained, the conversion rules of similar formats are used to reason and generate candidate conversion schemes. After testing and verification, the effective rules are added to the conversion rule base.
[0010] Furthermore, after the data format conversion is completed, the validity of the data will be verified through a dual verification mechanism, including comparing the structure definition of the converted data with that of the target format, extracting key fields for consistency comparison with the source data, and if the verification fails, the conversion process log will be traced back to locate the erroneous data and perform automatic re-conversion.
[0011] Preferably, the generation of the parameter adjustment scheme includes: Collect hardware parameters, software configurations, and performance metrics of historical processing data for each computing platform, and set parameter synchronization trigger conditions; Parse the algorithm task input by the user, extract the algorithm's computational complexity, data volume requirements, and parallel processing capabilities, and obtain the parameter correlation of the algorithm task; A global parameter adjustment scheme is generated by combining the parameter synchronization triggering conditions and the parameter correlation of the algorithm task.
[0012] Preferably, after the structured task description file is generated, it is decomposed into multiple levels according to the task's time constraints, resource requirements, and dependencies. The complex task is broken down into atomic subtasks, and the execution order is allocated based on a priority algorithm. High-priority tasks occupy idle resources first, and low-priority tasks are dynamically inserted for execution when resources are idle.
[0013] A cross-platform adaptive data processing system includes: The interface encapsulation module is used to parse the native interfaces of each computing platform, extract the functional parameter information of the interfaces, and generate encapsulated standardized interfaces. The data transmission module is used to convert the communication protocols of different platforms into a unified communication protocol within the system according to the needs of data processing tasks, and to establish a data transmission channel. The format conversion module converts the data format based on format conversion rules, and after conversion, it transmits the data to the target platform through the established data transmission channel. The parameter configuration module is used to generate parameter adjustment schemes based on the characteristics of the target platform and the algorithm requirements, and send the parameter adjustment schemes to the corresponding computing platform. The computing platform then configures the parameters according to the parameter adjustment schemes. The structure description module is used to input data information for data processing tasks through a graphical interface, parse the input information, and generate a structured task description file.
[0014] Preferably, the structure description module includes an error display module, which is used to track the data processing progress of each computing platform in real time and display the progress information to the user in real time through a graphical interface. When all processing tasks are completed, the result integration unit integrates the processing results of each platform and provides feedback to the user on the graphical interface. If an error occurs during the processing, the error prompting unit will display the error information to the user in a timely manner and provide corresponding solutions.
[0015] Compared to existing technologies, the beneficial effects of this application are as follows: (1) This application standardizes the native interfaces of various computing platforms through interface encapsulation and protocol conversion via the cross-platform adaptation layer. Users do not need to master multiple programming languages and be familiar with the native interfaces of different platforms. They only need to call the functions of each platform through the unified standardized interface. The automated collaboration layer realizes the automatic establishment of data transmission channels and the automatic conversion of data formats, replacing the manual script writing or manual processing method, greatly reducing the extra coding burden of users in cross-platform collaboration, and avoiding errors that are easy to occur in manual operation. (2) This application reduces the waiting time and manual intervention of cross-platform data interaction through intelligent data transmission management and format conversion, making the data flow between platforms smoother and more efficient. The graphical interface makes task configuration, progress monitoring and result viewing more intuitive and convenient, shortens the task configuration time and result acquisition cycle, and improves the overall efficiency of cross-platform data processing. (3) By analyzing platform characteristics and algorithm requirements, this application automatically adjusts the calculation parameters, which can specifically avoid the performance bottleneck caused by platform limitations, while reducing the performance decline caused by user code problems. Based on historical performance data and optimization models, it achieves more accurate parameter configuration, enabling the algorithm to perform more ideally on different platforms and improving the effect and stability of data processing. (4) The interface encapsulation and protocol conversion mechanism of this application has good scalability. When a new computing platform is added, it is only necessary to standardize the encapsulation and protocol adaptation of its native interface to quickly incorporate it into the system to achieve collaborative work. There is no need to make significant adjustments to the overall system architecture, which improves the system's adaptability to different computing platforms. Attached Figure Description
[0016] In the attached diagram: Figure 1 This is a schematic diagram of the method steps in an embodiment of this application; Figure 2 This is a schematic diagram of the system structure according to an embodiment of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments. Generally, the components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Example
[0018] like Figure 1 As shown, a cross-platform adaptive data processing method includes the following steps: The native interfaces of each computing platform are parsed, the functional parameter information of the interfaces is extracted, and a standardized encapsulated interface is generated. The generation of standardized interface encapsulation includes: Version tracing is performed on the parsed native interface function parameter information to extract the interface release time, update logs, and version iteration differences, and an interface version mapping table is constructed. The differences in functional parameters between different versions are matched a second time with the preset standardized interface template, and the functions of the native interface are mapped to the corresponding modules of the standardized interface to generate the encapsulated standardized interface.
[0019] After the standardized interface is encapsulated and generated, the interface access permission configuration of each computing platform is parsed, a permission mapping matrix is established, the platform's native permission rules are converted into a unified system permission identification system, the permission level of the request initiator is automatically verified when the interface is called, a standardized permission insufficient response is returned for call requests that exceed the permission scope, and permission verification logs are recorded.
[0020] Based on the requirements of data processing tasks, the communication protocols of different platforms are converted into a unified communication protocol within the system to establish a data transmission channel; Communication protocol conversion includes: It monitors communication requests between various computing platforms and the system in real time. When it receives a request for a data processing task, it parses the communication protocol type carried in the request and converts the communication protocols of different platforms into a unified communication protocol within the system according to the mapping relationship in the preset protocol conversion rules. When there is a many-to-one mapping conflict in the preset protocol conversion rules, the contextual semantics of the parsed fields are analyzed, and the optimal mapping rule is selected by combining the success rate statistics of historical conversions.
[0021] The data format is converted based on the format conversion rules, and after the conversion is completed, it is transmitted to the target platform through the established data transmission channel; The format conversion rules are pre-stored in the rule base, which contains information on the data formats supported by each computing platform and the conversion rules between different formats. At the same time, historical cases of format conversion are collected, and the combination of formats not covered is identified through cluster analysis. When a new format conversion requirement is obtained, the conversion rules of similar formats are used to reason and generate candidate conversion schemes. After testing and verification, the effective rules are added to the conversion rule base.
[0022] After the data format conversion is completed, the validity of the data will be verified through a dual verification mechanism, including comparing the structure definition of the converted data with that of the target format, extracting key fields for consistency comparison with the source data, and if the verification fails, the conversion process log will be traced back to locate the erroneous data and perform automatic reconversion.
[0023] Based on the characteristics of the target platform and the algorithm requirements, a parameter adjustment plan is generated and sent to the corresponding computing platform. The computing platform then configures the parameters according to the parameter adjustment plan. The generation of the parameter adjustment scheme includes: Collect hardware parameters, software configurations, and performance metrics of historical processing data for each computing platform, and set parameter synchronization trigger conditions; Parse the algorithm task input by the user, extract the algorithm's computational complexity, data volume requirements, and parallel processing capabilities, and obtain the parameter correlation of the algorithm task; A global parameter adjustment scheme is generated by combining the parameter synchronization triggering conditions and the parameter correlation of the algorithm task.
[0024] Users can input relevant information about the data processing task through a graphical interface, and the system will parse the user input to generate a structured task description file. After the structured task description file is generated, it is decomposed into multiple levels according to the task's time constraints, resource requirements, and dependencies. Complex tasks are broken down into atomic subtasks, and the execution order is allocated based on a priority algorithm. High-priority tasks occupy idle resources first, while low-priority tasks are dynamically inserted for execution when resources are idle. Example
[0025] The cross-platform adaptive data processing methodology was applied to an enterprise data center, integrating Windows servers, Linux clusters, and Mac workstations. The detailed implementation process is as follows: For the Windows Server 2019 platform, the Windows API Monitor tool was used to scan system libraries such as kernel32.dll and extract parameter information of the file operation interface Create File, including 23 functional parameters such as lp File Name and dw Desired Access. For Linux CentOS 7 clusters, the readelf command is used to parse the open system call interface in libc.so, obtain 18 parameters such as pathname and flags, and synchronously record the physical location of the interface in the / usr / lib64 directory. For the macOS Monterey workstation, the open interface in libSystem.B.dylib was analyzed using the otool tool in Xcode, and 21 parameters such as path and oflag, as well as information about the libdispatch library that the interface depends on, were extracted. According to Microsoft MSDN documentation, the CreateFile interface added the FILE_FLAG_OPEN_REPARSE_POINT parameter in Windows Server 2012, and the asynchronous I / O performance parameters were optimized in version 2019. The Linux kernel changelog shows that the open interface added the O_TMPFILE flag in version 3.10. Build a version mapping table, for example: Platform | Interface Name | V1.0 (2012) | V2.0 (2019) | |Windows|CreateFile|No FILE_FLAG_OPEN_REPARSE_POINT|Includes this parameter| The Windows dwDesiredAccess parameter is matched with the access_control module of the standardized interface, and the Linux flags parameter is mapped to the same module, thus completing the encapsulation. Parse the Windows ACL permission configuration and convert the GENERIC_ALL permission of the "Administrator" role to the system-wide unified "P0 level" permission identifier; in Linux rwx permissions, the root user's rwxrwxrwx is converted to "P0 level", and the user group's rwx------ is converted to "P1 level". Establish a permission mapping matrix. When it detects that a macOS user is attempting to call an interface requiring "P0 level" permission but actually has "P2 level" permission, return a standardized response message: {"code":403,"msg":"Insufficient permissions, P0 level permission required"}, and generate a log in the / var / log / permission directory. 2023-10-0114:30:user=test,interface=CreateFile,required=P0,actual=P2. The system deploys a protocol listening service. When the Windows platform sends a POST / data / upload request via the HTTP protocol, it parses the request header Content-Type: application / json and converts the HTTP protocol to the system's internal "DataBusv1.0" protocol according to preset rules. Example of conversion rules: HTTP header fields are mapped to DataBus Meta fields, and Body fields are mapped to Payload fields.
[0026] If a Linux platform sends the STOR / tmp / data.csv command via the FTP protocol, after parsing the protocol type, the FTP command-response mode is converted into the DataBus request-response mode, where the STOR command is mapped to the DataBus Upload opcode.
[0027] When both the HTTP Content-Length field and the FTP SIZE field are mapped to the DataBus data_size field, the parsing context is as follows: If the request comes from a file upload task, both Content-Length and SIZE represent the file size. Querying historical conversion records shows that the conversion success rate for this field is 98% for HTTP and 95% for FTP. Selecting the HTTP mapping rule, Content-Length: 1024 is converted to data_size: 1024.
[0028] The rule base pre-stores rules including: JSON and XML conversion rules (JSON {"name":"a"} converted to XML). <name> a< / name> ), CSV and Parquet conversion rules, mapping field name rows to Parquet Schema. Historical case studies revealed a missing conversion rule for "JSON array to XML nested node," which was identified through cluster analysis. When new requirements arise, candidate solutions are generated based on the "JSON object to XML node" conversion rule: [{"id":1},{"id":2}]→ <root><itemid="1" / ><itemid="2" / >< / root> After testing with 1000 samples and verifying a 99% success rate, it was added to the rule base. When the converted data is in XML format, compare its structure to the target schema definition: xs:elementname="root"type="rootType". If it is missing... <root>The tag verification failed. Extract the id field value "123" from the source JSON and compare it with the converted XML. <id> 123< / id> If the comparison is inconsistent, the conversion log is traced back to the format conversion error in step 5, and automatic reconversion is performed. Collected Windows server hardware parameters: CPU i9-12900K (16 cores), memory 64GB; Linux cluster: 20 nodes, each node CPU E5-2670 (8 cores), memory 32GB; Historical performance indicators: Windows takes an average of 8 minutes to process 10GB of data, while the Linux cluster takes 2 minutes to process in parallel. Set trigger conditions: When the task data volume is greater than 50GB, the parameter adjustment will be triggered. The user inputs a data mining algorithm task, which extracts data with a computational complexity of O(n²), requires 100GB of data, and supports 8-way parallelism. Based on the triggering conditions (100GB > 50GB) and parameter correlation, the parallelism is positively correlated with the number of CPU cores. The generated scheme is as follows: Windows server with 12 CPU cores and 48GB of memory; Linux cluster with 10 nodes, each with 6 CPU cores and 24GB of memory, and the parallelism of the algorithm is set to 18 (12+6). Users input the following through a graphical interface: Task name "User Behavior Analysis", Time constraint "Complete within 2 hours", Resource requirement "At least 8 CPU cores", Dependency "Data cleaning task completed". The system parses and generates structured files: { "task_id":"T001", "name":"User Behavior Analysis", "time_limit":"7200s", "resources":{"cpu_cores":8}, "dependencies":["T000"] } It is decomposed into 3 atomic subtasks: data extraction (T001-1), feature engineering (T001-2), and model training (T001-3). Based on the priority algorithm, model training (high priority) will take priority to occupy the idle 8-core CPU of the Windows server, while data extraction (low priority) will be inserted and executed when the Linux cluster nodes are idle. When high-priority task T001-3 occupies resources, low-priority task T001-1 enters the waiting queue; when two nodes in the Linux cluster are detected to be idle (each with 4 CPU cores), T001-1 is immediately triggered to execute; after T001-3 completes, the released resources are allocated to T001-2 to continue execution. Example
[0029] like Figure 2 As shown, a cross-platform adaptive data processing system includes: The interface encapsulation module is used to parse the native interfaces of each computing platform, extract the functional parameter information of the interfaces, and generate encapsulated standardized interfaces. The data transmission module is used to convert the communication protocols of different platforms into a unified communication protocol within the system according to the needs of data processing tasks, and to establish a data transmission channel. The format conversion module converts the data format based on format conversion rules, and after conversion, it transmits the data to the target platform through the established data transmission channel. The parameter configuration module is used to generate parameter adjustment schemes based on the characteristics of the target platform and the algorithm requirements, and send the parameter adjustment schemes to the corresponding computing platform. The computing platform then configures the parameters according to the parameter adjustment schemes. The structure description module is used to input data information for data processing tasks through a graphical interface, parse the input information, and generate a structured task description file.
[0030] The structure description module includes an error display module, which tracks the data processing progress of each computing platform in real time and displays the progress information to the user through a graphical interface. When all processing tasks are completed, the result integration unit integrates the processing results of each platform and provides feedback to the user on the graphical interface. If an error occurs during the processing, the error prompting unit will promptly display the error information to the user and provide corresponding solutions.
[0031] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications, improvements, and substitutions without departing from the concept of this application, and these all fall within the protection scope of this application.< / root>
Claims
1. A cross-platform adaptive data processing working method, characterized in that, The method comprises the following steps: Parsing the native interface of each computing platform, extracting the function parameter information of the interface, and generating a packaged standardized interface; According to the requirements of the data processing task, the communication protocols of different platforms are converted into a unified communication protocol within the system, and a data transmission channel is established; Based on the format conversion rule, the data format is converted, and after the conversion is completed, it is transmitted to the target platform through the established data transmission channel; According to the characteristics and algorithm requirements of the target platform, a parameter adjustment scheme is generated, and the parameter adjustment scheme is sent to the corresponding computing platform, and the computing platform configures parameters according to the parameter adjustment scheme; Through the graphical interface, input the data information of the data processing task, analyze the input information, and generate a structured task description file.
2. The cross-platform adaptive data processing working method according to claim 1, characterized in that, The packaging generation of the standardized interface comprises: Version tracing is performed on the parsed native interface function parameter information, the interface release time, update log and version iteration difference are extracted, and an interface version mapping table is constructed, The difference items of different versions of function parameters are matched with the preset standardized interface template, the functions of the native interface are mapped to the corresponding modules of the standardized interface, and the packaged standardized interface is generated.
3. The cross-platform adaptive data processing working method according to claim 2, characterized in that, After the standardized interface is packaged, the interface access permission configuration of each computing platform is parsed, a permission mapping matrix is established, the platform native permission rules are converted into a unified permission identification system of the system, and the permission level of the request initiator is automatically checked when the interface is called. The calling request beyond the permission range returns a standardized insufficient permission response, and the permission verification log is recorded.
4. The cross-platform adaptive data processing working method according to claim 1, characterized in that, The communication protocol conversion comprises: Real-time monitoring of communication requests between each computing platform and the system, when receiving the requirements of the data processing task, parsing the communication protocol type carried in the request, and converting the communication protocols of different platforms into a unified communication protocol within the system according to the mapping relationship in the preset protocol conversion rule; When there is a many-to-one mapping conflict in the preset protocol conversion rule, the context semantics of the field is parsed, the success rate of historical conversion is combined, and the optimal mapping rule is selected.
5. The cross-platform adaptive data processing working method according to claim 1, characterized in that, The format conversion rule is to store the data format information supported by each computing platform and the conversion rule between different formats in the rule library, and to collect historical cases of format conversion. Through clustering analysis, identify the uncovered format combination, when a new format conversion requirement is obtained, infer based on the conversion rule of similar formats, generate a candidate conversion scheme, and after testing and verification, supplement the effective rule to the conversion rule library.
6. The cross-platform adaptive data processing working method according to claim 5, characterized in that, After the data format conversion is completed, the data validity is verified through a double-checking mechanism, including comparing the converted data with the structure definition of the target format, extracting the key field and the source data for consistency comparison, if the verification fails, backtrack the conversion process log, locate the error data and perform automatic re-conversion.
7. The cross-platform adaptive data processing working method according to claim 1, characterized in that, The generation of the parameter adjustment scheme comprises: Collecting the hardware parameters, software configuration and historical processing data performance indicators of each computing platform, and setting parameter synchronization trigger conditions; Parsing the algorithm task input by the user, extracting the algorithm complexity, data volume requirement and parallel processing capability, and obtaining the parameter correlation of the algorithm task; A global parameter adjustment scheme is generated in combination with a parameter synchronization trigger condition and parameter relevance of an algorithm task.
8. The cross-platform adaptive data processing working method according to claim 1, characterized in that, After the structured task description file is generated, the complex task is split into atomic subtasks according to time constraints, resource requirements and dependency relationships of the task, and an execution order is assigned based on a priority algorithm, in which high-priority tasks preferentially occupy idle resources, and low-priority tasks are dynamically inserted for execution when resources are idle.
9. A cross-platform adaptive data processing work system, characterized by, The method comprises the steps of: An interface packaging module is configured to analyze the native interface of each computing platform, extract functional parameter information of the interface, and generate a packaged standardized interface; A data transmission module is configured to convert the communication protocol of different platforms into a unified communication protocol within the system according to the requirements of the data processing task, and establish a data transmission channel; A format conversion module is configured to convert the data format based on format conversion rules, and transmit the converted data to the target platform through the established data transmission channel; A parameter configuration module is configured to generate a parameter adjustment scheme according to the characteristics of the target platform and the algorithm requirements, send the parameter adjustment scheme to the corresponding computing platform, and configure the parameters of the computing platform according to the parameter adjustment scheme; A structure description module is configured to input data information of the data processing task through a graphical interface, analyze the input information, and generate a structured task description file.
10. The cross-platform adaptive data processing work system of claim 9, wherein, The structure description module includes an error display module configured to track the data processing progress of each computing platform in real time, display the progress information to the user in real time through the graphical interface, integrate the processing results of each platform when all the processing tasks are completed, and feed back the results to the user on the graphical interface. If an error occurs during the processing, the error prompt unit will display the error information to the user in a timely manner and provide corresponding solutions.