Data co-processing method, system and equipment based on data processing capability

Abstract

The invention discloses a data co-processing method, system and equipment based on data processing capability, and the method comprises the steps: firstly collecting basic capability data to construct an initial credibility file, training a long and short-term memory network model to obtain an equipment capability evaluation module, and then generating an equipment future capability curve to judge the processing capability after inputting a co-processing task. According to the generation of a fragmentation strategy and a mapping table, through monitoring task execution, if equipment capability deviates, dynamic re-fragmentation is carried out, the mapping table is updated after integrity verification, finally an aggregation result is returned to a requester, and the credibility of the equipment and a capability history library are updated. According to the method, stable execution of the task is guaranteed by improving the equipment capability prediction accuracy and the task processing efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a data collaborative processing method, system and device based on data processing capability. Background Art

[0002] Against the backdrop of the rapid development of big data and cloud computing technologies, data processing capabilities have become a key indicator of the competitiveness of enterprises and institutions. Existing data processing primarily relies on centralized data processing systems and distributed computing frameworks. Centralized data processing systems aggregate data to central servers for unified processing. While this approach facilitates management and maintenance, it presents data transmission bottlenecks. When the data scale is large, central servers are prone to problems such as excessive load and response delays. Distributed computing frameworks, such as Hadoop and Spark, can improve processing efficiency by processing data in parallel through clustered nodes. However, in cross-regional and cross-departmental data collaboration scenarios, the heterogeneous formats and protocol differences of different data sources make effective data integration difficult. Furthermore, data transmission and synchronization between nodes consume significant network resources, and there is a lack of effective management mechanisms for data processing priorities and dynamic resource allocation.

[0003] Furthermore, existing technologies often neglect data security and privacy protection during data processing, significantly increasing the risk of sensitive data leakage when multiple parties collaborate on data processing. With the widespread adoption of emerging technologies such as the Internet of Things and artificial intelligence, data processing scenarios are becoming increasingly complex, and the requirements for real-time, accurate, and secure data processing are constantly increasing. Existing data processing technologies are no longer able to meet the needs of efficient collaborative data processing. A collaborative data processing method with enhanced data processing capabilities is urgently needed to address these issues. Summary of the Invention

[0004] In order to solve the problems existing in the background technology, this patent application aims to propose a data collaborative processing method based on data processing capabilities, and through innovative technical means, solve the problem that existing data processing technologies are difficult to meet the needs of efficient collaborative processing of multi-source heterogeneous data.

[0005] A first aspect of the present application provides a data collaborative processing method based on data processing capabilities, comprising: Collecting basic capability data of the device and constructing an initial trust profile of the device based on the basic capability data, so as to train a long short-term memory network model using the initial trust profile and historical capability data to obtain a device capability evaluation module; When a collaborative task is input, the device capability assessment module is called to generate a future capability curve for each device based on the device's historical capability data and current status data. The device's data processing capability is then determined based on the future capability curve and data lineage tracking. Generate a sharding strategy and a device sharding mapping table based on the device's trustworthiness, device data processing capabilities, and the future capability curve of each device; The edge coordination node monitors the task execution status. If it detects that the device capability deviates from the predicted value, it triggers a dynamic resharding operation. During the dynamic resharding process, the edge coordination node verifies the integrity of the shard and regenerates the target device shard mapping table based on the device's current capability and trustworthiness. After aggregating the task execution results, the edge coordination node returns them to the requester and records the collaboration data. It then updates the device’s trust and capability history library based on the recorded collaboration data.

[0006] Optionally, data lineage tracking and the future capability curve of each device are used to collaboratively process the data allocated to each device, including: Send data sample packets to each device, use data lineage tracking to obtain the data processing method of each device, and then obtain the data processing capability of each device; Based on the data processing capability of each device, the future capability curve of each device and the trustworthiness of the device, appropriate data is allocated to each device for collaborative processing of data shards.

[0007] Optionally, the trustworthiness of the device is generated by the historical trustworthiness of a single device and the affinity of the device, as follows: ； in, is the trustworthiness of the device, 、 is the adaptive adjustment factor for the current network environment, is the historical trust of a single device, Q is the affinity of the device, and A is the CPU usage of the current device.

[0008] Optionally, if it is detected that the device capability deviates from the predicted value, triggering a dynamic resharding operation includes: Mark abnormal points in data slices, select highly trusted devices from the backup device pool to take over the task, re-encrypt the slices and distribute them, and update the mapping table.

[0009] Optionally, it is characterized in that the collection of the basic capability data includes hardware configuration: processor performance, memory capacity, storage capacity, and network connection status: bandwidth, delay.

[0010] Optionally, in the dynamic resharding operation, situations in which device capabilities deviate from predicted values include sudden high latency and a significant drop in processing speed.

[0011] A second aspect of the present application provides a data collaborative processing system based on data processing capabilities, comprising: The model training module is used to deploy the long-short-term memory network model and use the historical capability data of the equipment to train the equipment capability assessment module; The capability prediction module is used to call the trained device capability assessment module after inputting the collaborative task, generate the future capability curve of each device, and judge the device data processing capability based on the future capability curve of each device and data lineage tracking; The task allocation module is used to generate a sharding strategy based on the trust level of the device, the device data processing capability, and the future capability curve of each device, and to generate a device sharding mapping table for data sharding; The monitoring and resharding module is used to monitor the task execution status through the edge coordination node. When it detects that the device capability deviates from the predicted value, it triggers the dynamic resharding operation and verifies the integrity of the sharding. The data processing and update module is used to aggregate task execution results and return them to the requester, while recording collaborative data and updating the device's trust and capability history library.

[0012] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data collaborative processing method based on data processing capability when executing the computer program.

[0013] Compared with the prior art, the present invention has the following beneficial effects: The above-mentioned data collaborative processing method, system and device based on data processing capability collect basic capability data of the device and construct an initial trust profile of the device based on the basic capability data, so as to train the long-short-term memory network model using the initial trust profile and historical capability data to obtain a device capability evaluation module; when a collaborative task is input, the device capability evaluation module is called to generate a future capability curve for each device based on the historical capability data and current status data of the device, and the device data processing capability is determined based on the future capability curve of each device and data lineage tracking; based on the trust of the device, the device data processing capability and the future capability curve of each device, a sharding strategy is generated, and a device sharding mapping table is generated to perform data sharding; the task execution status is monitored by the edge coordination node, and if it is detected that the device capability deviates from the predicted value, a dynamic resharding operation is triggered, so that during the dynamic resharding process, the edge coordination node verifies the sharding integrity and regenerates the target device sharding mapping table based on the current capability and trust of the device; the edge coordination node returns the aggregated task execution results to the requester, records the collaborative data of this time, and updates the device trust and capability history library based on the recorded collaborative data. This application ensures stable task execution by improving the accuracy of equipment capability prediction and task processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of a data collaborative processing method based on data processing capability in one embodiment of the present invention; Figure 2 is a schematic diagram of a data collaborative processing system based on data processing capabilities in one embodiment of the present invention; Figure 3 It is a schematic diagram of a robot production safety supervision device in one embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0016] In one embodiment, if Figure 1 As shown, a data collaborative processing method based on data processing capability is provided. Figure 1 Take the example of , and explain it, including the following specific steps: S10: Collect basic capability data of the device, and build an initial trust profile of the device based on the basic capability data, so as to train a long short-term memory network model using the initial trust profile and historical capability data to obtain a device capability evaluation module.

[0017] Specifically, in this application, basic capability data of the device is collected, including the computing performance, storage capacity, network bandwidth, etc. of the device. These data constitute the basic attributes of the device. Based on these basic capability data, an initial trust profile of the device is constructed. The initial trust profile is an assessment of the device's ability to process data in its initial state. Then, combined with the device's historical capability data, that is, the device's past performance data in data processing tasks, a long short-term memory network model is used for training. The long short-term memory network model is good at processing sequence data. By learning the patterns in historical data, a device capability evaluation module is trained to predict the device's future data processing capabilities.

[0018] S20: After the collaborative task is input, the device capability evaluation module is called to generate the future capability curve of each device according to the historical capability data and current status data of the device, and the data processing capability of the device is determined based on the future capability curve of each device and data lineage tracking.

[0019] Specifically, when a collaborative task is input, the device capability assessment module is invoked. Based on the device's historical capability data and current status data (such as load and resource utilization), the module generates a future capability curve for each device. This curve, along with data lineage tracking, is used to determine the device's data processing capacity. Using these curves and data lineage tracking, the device's data processing capacity trends over a pre-set time period can be intuitively determined, providing a basis for task allocation.

[0020] Based on the device's trustworthiness, data processing capabilities, and each device's future capability profile, a sharding strategy is generated, resulting in a device sharding mapping table. This table clearly defines the data shards each device is responsible for processing, ensuring a reasonable distribution of tasks. The device's trustworthiness changes dynamically based on its performance during task execution. The sharding strategy is a rule for sharding based on data volume, data type, and other factors.

[0021] For example, collaborative task data is received based on collaborative task requests submitted by users, such as large-scale data analysis and cross-network file synchronization, and then task requirements are parsed, such as data volume, real-time requirements, and security levels. According to the user's collaborative task, the current capability indicators of all devices participating in data collaborative processing are collected in real time, such as the physical and network conditions of the devices such as CPU occupancy, network latency, and remaining memory. That is, after judging the data processing capabilities of each device involved in the data collaborative processing of the user's collaborative task, the collaborative task submitted by the user is input into the long-short-term memory network model to call the device capability evaluation module. The device capability evaluation module marks each device based on the device's historical capability data and current status data and generates a future capability curve for each device. The data processing capability is evaluated using the future capability curve of each device as a reference object. After comprehensively evaluating each device in combination with other conditions, a device sharding mapping table is generated.

[0022] The trustworthiness of a device is generated by combining the historical trustworthiness of each device and its affinity. The historical trustworthiness of a device is a quantitative evaluation indicator of its performance in executing tasks in past related activities. Based on the historical trustworthiness of a device, a comprehensive picture of its reliability, stability, and degree of compliance with expected performance in completing tasks can be obtained. The affinity of a device can reflect the level of coordination, difficulty, and adaptability of different devices and data when performing collaborative tasks. Device affinity depends not only on the hardware characteristics and functions of the device, but also on its collaborative capabilities in software, network environment, and task allocation mechanisms.

[0023] The affinity of the device is generated by the processing speed of the data category and data type in different devices, wherein the data category and data type are used to establish a data judgment mechanism in the process of new data processing and construct a dual-channel attention network, including a type attention channel and a category attention channel. The type attention channel: matches the processor architecture of different devices based on the data type (such as text, image, time series data) of the data in the collaborative task request for data processing. The category attention channel: uses a support vector machine to model the relationship between different data structures and different device processors based on the data structure (such as arrays, matrices and sets) of the data in the user's collaborative task request, and matches the data structure and processor type.

[0024] The trustworthiness of each device is obtained as follows: ； in, is the trustworthiness of the device, α，β is the adaptive adjustment factor for the current network environment, Tstatic The historical trustworthiness of a single device, Q is the affinity of the device, A The current CPU usage of the device.

[0025] Data from collaborative tasks submitted by users is sharded, taking into account factors such as the trust level and the pre-trained device capability assessment module. Data sharding involves dividing the original complete dataset into multiple independent yet interrelated sub-datasets, each of which is a shard. These shards can be independently stored, transmitted, and processed, and the processing results of each shard are then integrated to achieve operational objectives for the entire dataset.

[0026] It's worth noting that during data sharding, a small amount of redundancy or no redundancy is added when assigning large shards to high-trust, high-capability devices. Smaller shards are assigned to high-trust, low-capability devices with a small amount of redundancy or no redundancy. High trustworthiness indicates that the device has a reliable history of completing tasks on time, accurately, and rarely encountering errors or exceptions. High capability indicates that the device possesses powerful hardware and efficient processing capabilities, such as a high-performance processor, large memory capacity, and fast network connections. Based on these characteristics, large shards (e.g., a single shard accounting for 30% of the total) are assigned to these devices, effectively handing over larger task data volumes to these devices. The core logic behind this approach is to fully utilize their powerful processing capabilities, reduce the number of task shards, and, consequently, minimize the number of data transfers between devices. Because data transmission can be subject to network latency and packet loss, fewer transfers improve overall task efficiency. However, a certain amount of redundancy is added when assigning large shards to low-trust, high-capability devices, while redundancy is added when assigning small shards to low-trust, high-capability devices to ensure data security. Low-trust devices have historically exhibited unstable performance, prone to errors, or failures during task execution. To ensure data security, these devices are assigned redundant shards using Reed-Solomon erasure coding with a 50% redundancy setting. Reed-Solomon erasure coding is a coding method that enables recovery from data loss or corruption. It encodes the original data to generate data blocks containing redundant information. For example, if the original data consists of 10 units, using Reed-Solomon erasure coding with 50% redundancy will generate a total of 15 data blocks (10 original data blocks and 5 redundant data blocks). If data is lost or corrupted during processing on a low-trust device, this redundant information allows the system to recover the entire original data, thus ensuring data security and integrity.

[0027] After the data is sharded, the encryption algorithm is adjusted according to the real-time computing power of each device. On devices with sufficient computing power, that is, devices with CPU usage <60%, AES-256 encryption is used. On devices with limited computing power, that is, CPU usage ≥60%, Chacha20 lightweight encryption is switched to reduce the computing load. When encrypting the original data, the data is first encoded. The encoding is performed through the Transform model, and the input data is divided into multiple paragraphs according to the specified length. Each paragraph is encoded through the Transform model, and then an association is established between each segment of data and the two previous and next segments of data. When reading any paragraph, it is determined whether the previous and next segments are correct. The advantage of this is that data errors can be avoided for the sliced data, and the data integrity is checked after transmission to ensure that the data is correct and has not been attacked or damaged during transmission.

[0028] After data slices are generated, they are bound to the devices that process them, and a device shard mapping table is created, annotating the shard size, encryption method, and priority. In multiple network domains, an edge coordination node is elected based on device trust and network quality to be responsible for shard scheduling and result aggregation.

[0029] S30: Generate a sharding strategy based on the trust level of the device, the data processing capability of the device, and the future capability curve of each device, and generate a device sharding mapping table to perform data sharding.

[0030] The sharding strategy involves selecting a small, representative portion of the total data to be processed before a collaborative data task begins. This data is then assembled into a data sample package. This sample package encompasses data of varying types and characteristics, reflecting the overall data distribution and characteristics. The system then sends these sample packages to each participating device, ensuring they gain initial exposure to and experience with the data before formally processing large amounts of data.

[0031] As devices receive and process data packets, the system generates a unique identifier for each data packet and records information such as the processing path, method, and time taken on each device. Data lineage tracking allows the system to clearly understand the flow and changes of data between devices, providing a basis for subsequent analysis and decision-making. Data lineage tracking is a technology that records and tracks the entire process of data generation, processing, and final results.

[0032] After receiving a data sample, each device processes it according to its own data processing logic and algorithms. During processing, the device classifies the data based on, for example, type (text, image, numerical value), source, and characteristics. The system collects information on each device's processing of the data sample, including processing speed, accuracy, and the rationality of data classification. The system comprehensively evaluates each device's capabilities and strengths in processing different types of data by analyzing its future capability profile, device trustworthiness, and sample processing results.

[0033] Based on an analysis of each device's data sample processing, the system assigns appropriate data to each device for collaborative data processing based on its capabilities and strengths, task requirements, and data characteristics. Devices that excel in a particular type of data processing, with high speed and accuracy, are allocated more data of that type. Devices with relatively weaker processing capabilities or deficiencies in processing certain data types are allocated data volumes and types that match their capabilities, achieving efficient and collaborative data processing.

[0034] For example, in a user behavior data analysis task on a large e-commerce platform, massive amounts of user browsing history, purchase history, and comment data need to be analyzed to uncover consumer preferences and behavioral patterns. Devices A, B, and C participate in this collaborative data processing task. The system selects 1,000 representative data records from the user behavior data to be processed and compiles them into a data sample package. These data records include browsing, purchase, and comment data from different time periods, user groups, and product categories, encompassing the overall data diversity. The system then sends this data sample package to devices A, B, and C. The system generates a unique identifier for each data record in the data sample package and, during data transmission and processing, records the data processing path on devices A, B, and C. For example, this records information such as when the data arrived at the device, the processing steps it underwent on the device, and the algorithms invoked. After receiving the sample data package, device A uses a powerful text analysis algorithm to conduct an in-depth analysis of the review data, quickly and accurately classifying the reviews by sentiment (positive, negative, neutral). While processing data quickly, device A is slower and less accurate when processing image data (such as user-uploaded product images). Device B excels at processing numerical data. It performs statistical analysis on data such as purchase amounts and quantities in user purchase records, quickly and accurately calculating metrics such as total sales and average purchase quantity for each product category. However, it is less efficient when processing text data. Device C excels at image data processing, extracting features and classifying user-uploaded product images, but its processing capabilities are limited when processing large amounts of text and numerical data. The system collected the processing results of the sample data packages from devices A, B, and C, and concluded that device A excels in text sentiment classification, device B has a clear advantage in numerical statistical analysis, and device C excels in image data processing. The system allocated the data based on each device's future capability profile, its trustworthiness, and its processing capabilities and strengths in the sample data package. Approximately 70% of the review data was assigned to device A for sentiment classification; 60% of the numerical data from user purchase records was assigned to device B for statistical analysis; and 80% of the product image data uploaded by users was assigned to device C for feature extraction and classification. Furthermore, to ensure the coordination and integrity of data processing, data was rationally cross-distributed and verified, ensuring that the data processed by each device was interconnected and complementary, ultimately achieving efficient collaborative processing of user behavior data analysis tasks.

[0035] A device shard mapping table is generated according to the data sharding situation. The device shard mapping table is a data structure used to record the corresponding relationship between data shards and devices.

[0036] S40: Monitor the task execution status through the edge coordination node. If it is detected that the device capability deviates from the predicted value, the dynamic resharding operation is triggered. During the dynamic resharding process, the edge coordination node verifies the integrity of the shard and regenerates the target device shard mapping table based on the current capability and trust of the device.

[0037] Specifically, during task execution, the edge coordination node monitors the task execution status in real time. If it detects that device capabilities deviate from predicted values, such as performance degradation or device failure, a dynamic resharding operation is triggered. During dynamic resharding, the edge coordination node verifies the integrity of the shards to ensure data accuracy and consistency. It then regenerates the target device shard mapping table based on the device's current capabilities and trustworthiness to adapt to the change in device capabilities and ensure smooth task execution.

[0038] For example, in a large e-commerce platform's promotional order processing data collaboration task, the system deployed multiple edge devices to participate in order data processing, including order information parsing, inventory query, price calculation, etc. The edge coordination node monitors the operating status of each device in real time.

[0039] Device A was responsible for parsing order information for a specific region. Initially, Device A operated normally, completing the assigned tasks on time and efficiently, as predicted by the device capacity assessment module. However, as promotional activities gained momentum and order volume surged, Device A began to experience lag due to excessive memory usage, significantly reducing its data processing speed. Testing by the edge coordination node revealed that its actual processing capacity deviated significantly from the predicted value.

[0040] At this point, the edge coordination node immediately triggers a dynamic resharding operation. First, a data verification algorithm verifies the integrity of the order information shards processed by device A to ensure data accuracy. Next, based on the current operating status of other devices in the system (for example, devices B and C currently have low loads, strong processing capabilities, and high trustworthiness) and their real-time trustworthiness, a new task allocation plan is formulated, shifting the portion of the order information parsing task originally handled by device A to devices B and C.

[0041] After receiving the newly assigned task slices, devices B and C quickly began processing them. The edge coordination node continuously monitored the execution status of the new tasks to ensure smooth progress. Ultimately, thanks to the dynamic resharding mechanism, despite the performance degradation of device A, the entire order processing task was completed on time, ensuring the smooth progress of the promotion and guaranteeing the accuracy and consistency of order data processing.

[0042] In this embodiment, dynamic resharding can promptly respond to sudden performance degradation or failure of the device, avoiding stagnation or failure of the entire task due to a single device anomaly, thereby greatly enhancing the stability and reliability of the task execution process.

[0043] S50: The edge coordination node returns the aggregated task execution results to the requester, records the collaboration data, and updates the device's trust and capability history library based on the recorded collaboration data.

[0044] Specifically, after completing the aggregation of task execution results, the edge coordination node returns the results to the requester. At the same time, it records the collaborative data, including the device's task execution status and capability performance. Based on this record, the device's trust and capability history database is updated, providing more accurate reference data for subsequent task allocation and device capability prediction.

[0045] For example, assume that in a smart logistics scenario, there are multiple warehouses and distribution centers. Each warehouse and distribution center is equipped with data collection and processing equipment, including servers, smart terminals, etc. These devices need to work together to complete tasks such as order processing, inventory management, and route planning.

[0046] First, we collect basic capability data for each device, such as the number of CPU cores and memory size of servers, and the processing speed of smart terminals, to build an initial trust profile. We also collect historical capability data, such as the time and accuracy of each device's past logistics order processing, and train it using a long-short-term memory network model to develop a device capability assessment module.

[0047] When a new batch of logistics orders is placed, the historical order processing data of each device and the current load status of the device (such as the number of orders currently being processed and resource utilization) are input into the device capacity assessment module to generate the future order processing capacity curve of each device and judge the efficiency and capacity of each device in processing orders in the next period of time.

[0048] A device sharding mapping table is generated based on each device's trustworthiness (for example, devices with high previous order processing accuracy are more trusted), the sharding strategy (sharding by order receiving region), and the future order processing capacity curve. For example, orders from a certain region can be assigned to warehouse servers with strong processing capabilities and high trustworthiness.

[0049] During order processing, the edge coordination node monitors and detects a warehouse server failure, causing a decrease in processing capacity and deviation from the predicted value, triggering a dynamic resharding operation. After verifying the integrity of the processed order data, the edge coordination node regenerates the target device shard mapping table based on the current processing capacity and trustworthiness of other devices, and reallocates the order originally assigned to the failed server to other devices.

[0050] After each device completes order processing, the edge coordination node aggregates the processing results and returns them to the logistics management system. It also records relevant data about each device's order processing, such as processing time and accuracy, and updates the device's trust and capability history database to facilitate more reasonable allocation of order processing tasks.

[0051] In one embodiment, if Figure 2 As shown, a data collaborative processing system based on data processing capabilities is provided. The data collaborative processing system for data processing capabilities corresponds one-to-one to the data collaborative processing system method for data processing capabilities in the above embodiment. The data collaborative processing system for data processing capabilities includes: a data acquisition and trust building module, a model training module, a capability prediction module, a task allocation module, a monitoring and resharding module, and a data processing and update module. The functional modules are described in detail as follows: The data collection and trust building module is used to collect basic device capability data during device registration and build an initial trust profile for the device; The model training module is used to deploy the long-short-term memory network model and use the historical capability data of the equipment to train the equipment capability assessment module; The capability prediction module is used to call the trained equipment capability assessment module after inputting the collaborative task to generate the future capability curve of each equipment; The task allocation module is used to generate a sharding strategy and a device sharding mapping table based on the trust level of the device and the future capability curve of each device. The sharding strategy uses data lineage tracking and the future capability curve of each device to coordinate the data allocation of each device; The monitoring and resharding module is used to monitor the task execution status through the edge coordination node. When it detects that the device capability deviates from the predicted value, it triggers the dynamic resharding operation and verifies the integrity of the sharding. The data processing and update module is used to aggregate task execution results and return them to the requester, while recording collaborative data and updating the device's trust and capability history library.

[0052] For the specific definition of the data collaborative processing system of data processing capabilities, please refer to the definition of the data collaborative processing method of data processing capabilities above, which will not be repeated here. The various modules in the data collaborative processing system of data processing capabilities can be implemented in whole or in part by software, hardware, and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0053] In one embodiment, if Figure 3As shown, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, a data collaborative processing method based on data processing capability is implemented. For the specific limitations on electronic devices, please refer to the above limitations on the data collaborative processing method based on data processing capabilities, which will not be repeated here.

[0054] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM). Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0055] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data collaborative processing method based on data processing capability, characterized in that: include: Collecting basic capability data of the device and constructing an initial trust profile of the device based on the basic capability data, so as to train a long short-term memory network model using the initial trust profile and historical capability data to obtain a device capability evaluation module; When a collaborative task is input, the device capability assessment module is called to generate a future capability curve for each device based on the device's historical capability data and current status data. The device's data processing capability is then determined based on the future capability curve and data lineage tracking. Generate a sharding strategy based on the device's trustworthiness, device data processing capabilities, and each device's future capability curve, and generate a device sharding mapping table for data sharding. The edge coordination node monitors the task execution status. If it detects that the device capability deviates from the predicted value, it triggers a dynamic resharding operation. During the dynamic resharding process, the edge coordination node verifies the integrity of the shard and regenerates the target device shard mapping table based on the device's current capability and trustworthiness. After aggregating the task execution results, the edge coordination node returns them to the requester and records the collaboration data. It then updates the device’s trust and capability history library based on the recorded collaboration data.

2. The data collaborative processing method based on data processing capability according to claim 1, characterized in that: Using data lineage tracking and each device's future capability curve, data is allocated to each device for collaborative processing, including: Send data sample packets to each device, use data lineage tracking to obtain the data processing method of each device, and then obtain the data processing capability of each device; Based on the data processing capability of each device, the future capability curve of each device and the trustworthiness of the device, appropriate data is allocated to each device for collaborative processing of data shards.

3. The data collaborative processing method based on data processing capability according to claim 1, characterized in that: The trustworthiness of a device is generated by the historical trustworthiness of a single device and the affinity of the device, as follows: ； in, is the trustworthiness of the device, 、 is the adaptive adjustment factor for the current network environment, is the historical trustworthiness of a single device, The affinity of the device, The current CPU usage of the device.

4. The data collaborative processing method based on data processing capability according to claim 1, characterized in that: If it is detected that the device capability deviates from the predicted value, a dynamic resharding operation is triggered, including: Mark abnormal points in data slices, select highly trusted devices from the backup device pool to take over the task, re-encrypt the slices and distribute them, and update the mapping table.

5. The data collaborative processing method based on data processing capability according to claim 1, characterized in that: The collection of the basic capability data includes hardware configuration: processor performance, memory capacity, storage capacity, and network connection status: bandwidth, latency.

6. The data collaborative processing method based on data processing capability according to claim 1, characterized in that: In the dynamic resharding operation, situations where the device capability deviates from the predicted value include sudden delays and decreased processing speed.

7. A data collaborative processing system based on data processing capability, characterized in that: include: The data collection and trust building module is used to collect basic device capability data during device registration and build an initial trust profile for the device; The model training module is used to deploy the long-short-term memory network model and use the historical capability data of the equipment to train the equipment capability assessment module; The capability prediction module is used to call the trained device capability assessment module after inputting the collaborative task, generate the future capability curve of each device, and judge the device data processing capability based on the future capability curve of each device and data lineage tracking; The task allocation module is used to generate a sharding strategy based on the trust level of the device, the device data processing capability, and the future capability curve of each device, and to generate a device sharding mapping table for data sharding; The monitoring and resharding module is used to monitor the task execution status through the edge coordination node. When it detects that the device capability deviates from the predicted value, it triggers the dynamic resharding operation and verifies the integrity of the sharding. The data processing and update module is used to aggregate task execution results and return them to the requester, while recording collaborative data and updating the device's trust and capability history library.

8. The data collaborative processing system based on data processing capability according to claim 7, characterized in that: The basic capability data collected by the data collection and trust building module includes at least one of the hardware configuration of the device, the network connection status, and the historical task execution record.

9. The data collaborative processing system based on data processing capability according to claim 5, characterized in that: In the monitoring and resharding module, situations where the device capability deviates from the predicted value include sudden delays and decreased processing speed.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data collaborative processing method based on data processing capability according to any one of claims 1 to 6 when executing the computer program.

Images