A data collaborative processing method, system and device based on data processing capacity

Abstract

The application discloses a data collaborative processing method, system and device based on data processing capacity, constructs an initial trust degree archive by first collecting basic capacity data, trains a long short-term memory network model to obtain a device capacity evaluation module, then, after inputting a collaborative task, generates a future capacity curve of the device to judge processing capacity, generates a fragmentation strategy and a mapping table according to the future capacity curve, dynamically fragments again if the device capacity deviates through monitoring task execution, updates the mapping table after verifying integrity, and finally returns the aggregation result to a requester, and updates the trust degree and capacity history library of the device. The application improves the accuracy of device capacity prediction and the efficiency of task processing, and guarantees stable execution of the task.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a data collaborative processing method, system and device based on data processing capacity. BACKGROUND

[0002] Under the background of rapid development of big data and cloud computing technology, data processing capacity has become a key indicator for measuring the competitiveness of enterprises and institutions. Existing data processing mainly relies on centralized data processing systems and distributed computing frameworks. The centralized data processing system processes data uniformly by gathering data to the center server. Although the above-mentioned method is convenient for management and maintenance, there is a data transmission bottleneck. When the data scale is large, the center server is prone to high load, response delay and other problems; distributed computing frameworks such as Hadoop, Spark, etc. can improve processing efficiency by parallel processing of data through cluster nodes, but in the cross-regional and cross-department data collaborative scene, the format heterogeneity and protocol difference of different data sources make it difficult to effectively integrate data, and the data transmission and synchronization between nodes consume a lot of network resources, and there is a lack of effective management mechanism for data processing priority and resource dynamic allocation.

[0003] In addition, the existing technology often ignores data security and privacy protection in the data processing process. In the multi-party data collaborative processing, the risk of leakage of sensitive data increases significantly. With the popularization of emerging technologies such as Internet of Things and artificial intelligence, data processing scenarios are becoming more and more complex, and the real-time, accuracy and security requirements of data processing are constantly improving. The existing data processing technology has been difficult to meet the needs of efficient collaborative processing of data, and there is an urgent need for a data collaborative processing method based on data processing capacity to solve the above problems. SUMMARY

[0004] In order to solve the problems in the background art, the present application aims to provide a data collaborative processing method based on data processing capacity, which solves the problem that the existing data processing technology has been difficult to meet the needs of efficient collaborative processing of multi-source heterogeneous data through innovative technical means.

[0005] The first aspect of the present application provides a data collaborative processing method based on data processing capacity, comprising:

[0006] Collecting the basic capability data of the device, and constructing the initial trust degree profile of the device according to the basic capability data, to train the long short-term memory network model by using the initial trust degree profile and historical capability data, and obtain a device capability evaluation module;

[0007] When the collaborative task is input, the device capability evaluation module is called to generate future capability curves of each device according to historical capability data and current state data of the device, and the data processing capability of each device is determined according to the future capability curves of each device and data bloodline tracking;

[0008] According to the trust degree of the device, the data processing capability of the device and the future capability curves of each device, a sharding strategy is generated, and a device sharding mapping table is generated;

[0009] The edge coordination node monitors the task execution state, and if it is detected that the device capability deviates from the predicted value, a dynamic resharding operation is triggered, so that in the dynamic resharding process, the edge coordination node verifies the sharding integrity, and generates a target device sharding mapping table according to the current capability and trust degree of the device;

[0010] The edge coordination node returns to the requester after aggregating the task execution results, and records the collaborative data, and updates the trust degree and capability history library of the device according to the recorded collaborative data.

[0011] Optionally, the data of each device is allocated for collaborative processing using data bloodline tracking and future capability curves of each device, including:

[0012] Data sample packets are sent to each device, and the data processing method of each device is obtained using data bloodline tracking to obtain the data processing capability of each device;

[0013] According to the data processing capability of each device, the future capability curves of each device and the trust degree of the device, each device is allocated appropriate data for collaborative processing of data sharding.

[0014] Optionally, the trust degree of the device is generated by the historical trust degree of a single device and the affinity of the device, and specifically as follows: ；

[0015] Among them, the trust degree of the device, 、 is a current network environment adaptive adjustment factor, is the historical trust degree of a single device, Q is the affinity of the device, and A is the CPU occupancy rate of the current device.

[0016] Optionally, if the device capability deviates from the predicted value, a dynamic resharding operation is triggered, including:

[0017] The data slice abnormal point is marked as an abnormal shard, a high-trust-degree device is selected from a backup device pool to take over the task, the shard is re-encrypted and distributed, and the mapping table is updated.

[0018] Optionally, the collection of the basic capability data comprises hardware configuration: processor performance, memory capacity, storage capacity, and network connection status: bandwidth, delay.

[0019] Optionally, in the dynamic re-sharding operation, the case that the device capability deviates from the predicted value comprises a sudden high delay and a significant decrease in processing speed.

[0020] In a second aspect of the present application, a data collaborative processing system based on data processing capability is provided, comprising:

[0021] a model training module configured to deploy a long short-term memory network model and train a device capability evaluation module using historical capability data of the device;

[0022] a capability prediction module configured to, after inputting a collaborative task, call the trained device capability evaluation module to generate a future capability curve of each device and determine the data processing capability of the device according to the future capability curve of each device and data bloodline tracking;

[0023] a task allocation module configured to generate a sharding strategy and a device sharding mapping table according to the trust degree of the device, the data processing capability of the device, and the future capability curve of each device, and perform data sharding;

[0024] a monitoring and re-sharding module configured to monitor the task execution state through an edge coordination node, trigger a dynamic re-sharding operation when detecting that the device capability deviates from the predicted value, and verify the sharding integrity;

[0025] a data processing and updating module configured to aggregate the task execution results and return them to the requester, record the collaborative data, and update the trust degree of the device and the capability history library.

[0026] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data collaborative processing method based on data processing capability when executing the computer program.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] The data collaborative processing method, system and device based on data processing capability collect basic capability data of the device, and construct an initial trust degree profile of the device according to the basic capability data, so as to train a long short-term memory network model by using the initial trust degree profile and historical capability data to obtain a device capability evaluation module; when a collaborative task is input, the device capability evaluation module is called to generate future capability curves of each device according to historical capability data and current state data of the device, and to judge the data processing capability of the device according to the future capability curves of each device and data bloodline tracking; a sharding strategy is generated according to the trust degree of the device, the data processing capability of the device and the future capability curves of each device, and a device sharding mapping table is generated to perform data sharding; the edge coordination node monitors the task execution state, and if it is detected that the device capability deviates from the predicted value, a dynamic resharding operation is triggered, so that in the dynamic resharding process, the edge coordination node verifies the sharding integrity, and generates a target device sharding mapping table according to the current capability and trust degree of the device; the edge coordination node returns to the requester after aggregating the task execution result, and records the collaborative data, and updates the trust degree and capability history library of the device according to the recorded collaborative data. The application improves the accuracy of device capability prediction and the efficiency of task processing, and guarantees stable task execution. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flowchart of a data collaborative processing method based on data processing capability in an embodiment of the application;

[0030] Figure 2 is a schematic diagram of a data collaborative processing system based on data processing capability in an embodiment of the application;

[0031] Figure 3 is a schematic diagram of a robot production safety supervision device in an embodiment of the application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0033] In an embodiment, as shown in Figure 1 , a data collaborative processing method based on data processing capability is provided. Taking the application of the method in Figure 1 as an example, the following specific steps are described:

[0034] S10: Collecting basic capability data of the device, and constructing an initial trustworthiness profile of the device according to the basic capability data, so as to train a long short-term memory network model by using the initial trustworthiness profile and historical capability data, and obtain a device capability evaluation module.

[0035] Specifically, the basic capability data of the device is collected in the present application, including the computing performance, storage capacity, network bandwidth and the like of the device, which constitute the basic attributes of the device. Based on the basic capability data, an initial trustworthiness profile of the device is constructed, which is an evaluation of the data processing capability of the device in the initial state. Then, combined with the historical capability data of the device, i.e. the performance data of the device in the past data processing tasks, a long short-term memory network model is trained to obtain a device capability evaluation module, which is used to predict the future data processing capability of the device.

[0036] S20: When a collaborative task is input, the device capability evaluation module is called to generate future capability curves of each device according to the historical capability data and current state data of the device, and the data processing capability of the device is determined according to the future capability curves of each device and data bloodline tracking.

[0037] Specifically, when a collaborative task is input, the device capability evaluation module is called, and the device capability evaluation module generates future capability curves of each device according to the historical capability data and current state data of the device (such as the current load of the device, resource occupancy rate, etc.). The data processing capability of the device is determined according to the future capability curves of each device and data bloodline tracking. Through these curves and data bloodline tracking, the change trend of the data processing capability of the device in the future preset time period can be directly determined, which provides a basis for task allocation.

[0038] According to the trustworthiness of the device, the data processing capability of the device and the future capability curves of each device, a fragmentation strategy is generated after comprehensive consideration, and a device fragmentation mapping table is obtained. The device fragmentation mapping table clearly indicates the data fragments to be processed by each device, and realizes the reasonable allocation of tasks. The trustworthiness of the device will dynamically change with the performance of the device in the task execution process. The fragmentation strategy is a rule for fragmenting according to data volume, data type and the like.

[0039] Exemplary, according to the receiving user submitted collaborative task request to receive collaborative task data, such as large-scale data analysis, cross-network file synchronization, then, analyze the task demand, such as data volume, real-time requirement, security level. According to the user's collaborative task real-time acquisition of all participating data collaborative processing equipment current ability index, such as CPU occupancy, network delay, remaining memory and other equipment physical situation and network situation, that is, after judging the user's collaborative task participating data collaborative processing equipment processing data ability, the user submitted collaborative task input long short-term memory network model calls equipment capability evaluation module, equipment capability evaluation module according to the historical capability data and current state data of equipment, mark each device and generate each device future capability curve. Take each device future capability curve as the reference object to evaluate the data processing capacity. After comprehensive evaluation of each device combined with other conditions, generate device fragmentation mapping table.

[0040] The trust degree of the device is generated by the historical trust degree of each device and the affinity of the device. The historical trust degree of the device is a quantitative evaluation index of the performance of the device in the past related activities. According to the historical trust degree of the device, the reliability, stability and compliance with the expected performance of the device in the historical process can be obtained. The affinity of the device can reflect the cooperation of different devices and data in executing collaborative tasks, the difficulty and adaptation level of collaborative work. The affinity of the device not only depends on the hardware characteristics and functions of the device, but also involves the collaborative ability of the device in software layer, network environment and task allocation mechanism.

[0041] The affinity of the device is generated by the processing speed of the data category and the data type in different devices. The data category and the data type are used to establish a data judgment mechanism in the process of new data processing, and a double-channel attention network is constructed, including a type attention channel and a category attention channel. The type attention channel is based on the data type (such as text, image, time series data) of the data in the collaborative task request of data processing, and matches the processor architecture of different devices. The category attention channel is based on the data structure (such as array, matrix and set) of the data in the user's collaborative task request, and uses support vector machine to model the relationship between different data structures and different device processors, and matches the data structure and processor type.

[0042] The acquisition method of the trust degree of each device is: ；

[0043] Among them, is the trust degree of the device, α，β is the current network environment adaptive adjustment factor, Tstatica historical trust level of the individual device, Q an affinity level for the device, A a CPU occupancy rate of the current device.

[0044] Based on the trust level and other conditional elements such as the device capability evaluation module obtained through pre-training, the data in the collaborative task submitted by the user is data-fragmented. Data fragmentation is the process of dividing an original complete data set into multiple independent and interrelated sub-data sets, each of which is a fragment. These fragments can be independently stored, transmitted, and processed, and finally the processing results of each fragment are integrated to achieve the operation purpose of the entire data set.

[0045] It is worth noting that when data is fragmented, a small amount of redundancy or no redundancy is added when assigning large fragments to high-trust high-capability devices. A small amount of redundancy or no redundancy is added when assigning small fragments to high-trust low-capability devices. The high trust level means that the device has performed reliably in the past task execution, can complete the task on time and accurately, and rarely has errors or exceptions; high capability indicates that the device has strong hardware performance and high processing power, such as high-performance processors, large-capacity memories, fast network connections, etc. Based on these characteristics, large fragments (such as single fragments accounting for 30% of the total) are assigned to them, that is, larger-scale task data is given to such devices for processing. The core logic of this is to fully utilize its powerful processing power, reduce the number of task fragments, and thus reduce the number of data transmissions between devices. Because there may be network delays, packet loss, and other problems during data transmission, the fewer the number of transmissions, the higher the overall efficiency of task execution. However, a certain amount of redundancy is added when assigning large fragments to low-trust high-capability devices. Redundancy is added when assigning small fragments to low-trust high-capability devices to ensure data security. Low-trust devices have unstable performance and are prone to errors or failures in past task execution. To ensure data security, redundant fragments are assigned to such devices, and Reed-Solomon error correction code technology is used to set a 50% redundancy level. Reed-Solomon error correction code is a coding method that can recover data when it is lost or damaged. By encoding the original data, it generates data blocks containing redundant information. For example, if the original data is 10 units, after using Reed-Solomon error correction code with a 50% redundancy level, a total of 15 units of data blocks (10 original data blocks and 5 redundant data blocks) are generated. When a low-trust device experiences data loss or damage during processing, the system can recover the complete original data using this redundant information, thereby ensuring the security and integrity of the data.

[0046] After the data slicing is completed, the encryption algorithm is adjusted according to the real-time computing power of each device. In devices with sufficient computing power, i.e., devices with CPU usage < 60%, AES-256 encryption is used. In devices with limited computing power, i.e., devices with CPU usage >= 60%, Chacha20 lightweight encryption is switched to, reducing the computational load. When encrypting the original data, the data is first encoded, and the encoding is performed by a Transform model, which splits the input data into multiple paragraphs according to a specified length. Each paragraph is then encoded by the Transform model, and the association between each piece of data and the two adjacent pieces of data is established. When reading any paragraph, it is determined whether the adjacent paragraphs are correct. The advantage of this is that the sliced data can avoid data errors, ensure data integrity after transmission, and ensure data correctness without being attacked or damaged during transmission.

[0047] When the data slice is generated, the data slice is bound to the device processing the data to generate a device slice mapping table and mark the slice size, encryption method, and priority. In multiple network domains, based on the trustworthiness of the device and the network quality, an edge coordination node is elected to be responsible for slice scheduling and result aggregation.

[0048] S30: According to the trustworthiness of the device, the data processing capability of the device, and the future capability curve of each device, a slicing strategy is generated, and a device slice mapping table is generated to slice the data.

[0049] The slicing strategy is to select a small part of data with representative from all the data to be processed before the data collaboration task starts, to form a data sample package. The sample package covers different types and characteristics of data, and can reflect the distribution and characteristics of the overall data. The system sends these sample packages to each device participating in collaborative processing to ensure that the device has a preliminary contact and processing experience with the data before formally processing large-scale data.

[0050] During the process of receiving and processing data sample packages by the device, the system generates a unique identifier for each data sample package and records the processing path, processing method, processing time, and other information of the data sample package on each device. Through data bloodline tracking, the system can clearly understand the flow and changes of data between different devices, providing a basis for subsequent analysis and decision-making. The data bloodline tracking is a technology that records and tracks the entire process of data from generation, processing to final result.

[0051] After receiving the data sample, each device processes the sample according to its own data processing logic and algorithm. During the processing, the device classifies the data, for example, according to the type of data (text, image, numerical value), the source of the data, the characteristics of the data, etc. The system collects the processing results of each device on the data sample, including processing speed, processing accuracy, rationality of data classification, etc. Through the future capability curve of each device, the trustworthiness of the device, and the processing results of the sample, the ability and advantage of each device in processing different types of data are comprehensively evaluated.

[0052] Based on the analysis of the processing of the data sample by each device, the system allocates appropriate data for each device for collaborative processing of data fragmentation according to the ability and advantage of the device, combined with the requirements of the task and the characteristics of the data. For devices that perform well in processing a certain type of data, have fast processing speed and high accuracy, more data of that type is allocated; for devices with relatively weak processing capacity or deficiencies in processing certain types of data, the amount and type of data that matches their capacity is allocated to achieve efficient collaboration in data processing.

[0053] Exemplarily, in a user behavior data analysis task of a large e-commerce platform, it is necessary to analyze a large amount of user browsing records, purchase records, comment data and the like to mine the consumption preferences and behavior patterns of users. Devices A, B and C in the system participate in the data collaborative processing task. The system selects 1000 representative data records from the user behavior data to be processed to form a data sample package. These data records contain browsing, purchase and comment data of different time periods, different user groups and different commodity categories, covering the diversity of the overall data. The system sends the data sample package to devices A, B and C respectively. The system generates a unique identifier for each data record in the data sample package, and records the processing path of the data on devices A, B and C during data transmission and processing. For example, record when the data arrives at the device, which processing steps it has gone through on the device, which algorithms have been called and the like. After receiving the data sample package, device A uses powerful text analysis algorithms to conduct in-depth analysis on the comment data, quickly and accurately classifies the comment data according to the sentiment tendency (positive, negative, neutral), and the processing speed is relatively fast, but when processing image data (such as user uploaded commodity picture related data), the processing speed is slow and the accuracy is not high. Device B is good at processing numerical data, it conducts statistical analysis on the consumption amount and purchase quantity and the like in the user purchase record, can quickly and accurately calculate the sales total, average purchase quantity and the like of various commodities, but the efficiency is low when processing text data. Device C performs well in image data processing, can extract features and classify user uploaded commodity pictures, but has limited processing capacity when processing a large amount of text and numerical data. The system collects the processing results of the data sample package by devices A, B and C, and analyzes that device A is excellent in text sentiment classification, device B is obvious in numerical statistical analysis, and device C is excellent in image data processing. According to the future capacity curve of each device, the trust degree of the device and the processing capacity and advantages of the sample package, the system allocates the overall data. About 70% of the comment data is allocated to device A for sentiment classification processing; 60% of the numerical data in the user purchase record is allocated to device B for statistical analysis; 80% of the user uploaded commodity picture data is allocated to device C for feature extraction and classification. At the same time, in order to ensure the collaboration and integrity of data processing, reasonable cross allocation and verification are also performed on the data to ensure that the data processed by each device can be associated and supplemented with each other, and finally the efficient collaborative processing of the user behavior data analysis task is realized.

[0054] A device shard mapping table is generated according to the data shard situation, and the device shard mapping table is a data structure for recording the correspondence between data shards and devices.

[0055] S40: The edge coordination node monitors the task execution state, and if it detects that the device capability deviates from the predicted value, it triggers a dynamic resharding operation, so that in the dynamic resharding process, the edge coordination node verifies the shard integrity, and generates a target device shard mapping table according to the current device capability and trustworthiness.

[0056] Specifically, during task execution, the edge coordination node monitors the task execution state in real time. If it detects that the device capability deviates from the predicted value, for example, the device has performance degradation, failure, etc., a dynamic resharding operation is triggered. In the dynamic resharding process, the edge coordination node verifies the shard integrity to ensure the accuracy and consistency of the data, and then generates a target device shard mapping table according to the current device capability and trustworthiness, to adapt to the changes in device capability and ensure smooth task execution.

[0057] For example, in a large e-commerce platform's promotion activity order processing data collaboration task, multiple edge devices are deployed to process order data, including order information parsing, inventory query, price calculation, etc. The edge coordination node monitors the running state of each device in real time.

[0058] Among them, device A is responsible for processing part of the order information parsing task in some areas. In the early stage of task execution, device A is running normally and can efficiently complete the assigned task on time according to the prediction of the device capability evaluation module. However, as the promotion activity becomes popular, the order volume increases sharply, and device A has a high memory occupation and appears to be stuck, with a significant decrease in data processing speed. The edge coordination node detects that its actual processing capability deviates significantly from the predicted value.

[0059] At this time, the edge coordination node immediately triggers a dynamic resharding operation. First, it verifies the integrity of the order information shard data processed by device A using a data verification algorithm to ensure data accuracy. Then, according to the running state of other devices in the current system (such as devices B and C, which have low load and strong processing capability, and high trustworthiness), and their real-time trustworthiness, it re-formulates the task allocation scheme, and transfers part of the order information parsing task originally processed by device A to devices B and C.

[0060] After receiving the newly allocated task shards, devices B and C quickly begin processing. The edge coordination node continuously monitors the new task execution state to ensure smooth task progress. Ultimately, under the action of the dynamic resharding mechanism, although device A has performance degradation problems, the entire order processing task is still completed on time, ensuring the normal progress of the promotion activity, and also ensuring the accuracy and consistency of order data processing.

[0061] In this embodiment, through dynamic re-sharding, the sudden performance decline or failure of the device can be timely addressed, avoiding the entire task from stalling or failing due to a single device anomaly, and greatly enhancing the stability and reliability of the task execution process.

[0062] S50: The edge coordination node returns to the requester after aggregating the task execution results, records the current collaboration data, and updates the trust degree and capability history library of the device according to the recorded collaboration data.

[0063] Specifically, after completing the aggregation of the task execution results, the edge coordination node returns the results to the requester. At the same time, it records the current collaboration data, including the task execution of the device, the capability performance, and other information, and updates the trust degree and capability history library of the device according to these records, providing more accurate reference data for subsequent task allocation and device capability prediction.

[0064] For example, assume that in a smart logistics scenario, there are multiple warehouses and distribution centers, each equipped with data collection and processing devices, including servers, intelligent terminals, etc., which need to collaborate to complete tasks such as order processing, inventory management, and path planning.

[0065] First, collect the basic capability data of each device, such as the number of CPU cores, memory size of the server, and processing speed of the intelligent terminal, to construct an initial trust degree profile. At the same time, collect historical capability data such as the time and accuracy of past processing of logistics order data, and use a long short-term memory network model to train a device capability evaluation module.

[0066] When a new batch of logistics orders is placed, input the historical processing order data of each device and the current load of the device (such as the number of orders currently being processed and resource occupancy) into the device capability evaluation module to generate a future processing order capability curve for each device, and determine the efficiency and capability of each device in processing orders in the next period of time.

[0067] According to the trust degree of each device (for example, the trust degree of a device with a high order processing accuracy in the past), the sharding strategy (according to the delivery area of the order), and the future processing order capability curve, a device sharding mapping table is generated. For example, orders in a certain area are allocated to a warehouse server with strong processing capability and high trust degree for processing.

[0068] During order processing, the edge coordination node monitors and finds that a warehouse server has a processing capability decline due to a fault, deviating from the predicted value, triggering a dynamic re-sharding operation. After verifying the integrity of the processed order data, the edge coordination node regenerates a target device sharding mapping table according to the current processing capability and trust degree of other devices, and reallocates the orders originally allocated to the faulty server to other devices.

[0069] After each device completes the order processing, the edge coordination node aggregates the processing results and returns them to the logistics management system. At the same time, the relevant data of each device processing the order in this time, such as processing time, accuracy, etc., are recorded, and the trust degree and ability history library of the device are updated, so as to more reasonably allocate order processing tasks in the future.

[0070] In an embodiment, as shown in Figure 2 A data processing capability-based data collaborative processing system is provided, which corresponds to the data processing capability-based data collaborative processing system method in the above embodiment. The data processing capability-based data collaborative processing system includes a data collection and trust degree construction module, a model training module, an ability prediction module, a task allocation module, a monitoring and re-sharding module, a data processing and updating module, and the details of each functional module are as follows:

[0071] The data collection and trust degree construction module is used to collect device basic capability data when the device is registered, and to construct the initial trust degree profile of the device.

[0072] The model training module is used to deploy a long short-term memory network model, and to train the device capability evaluation module using historical capability data of the device.

[0073] The ability prediction module is used to generate future capability curves of each device after inputting a collaborative task, by calling the trained device capability evaluation module.

[0074] The task allocation module is used to generate a sharding strategy and a device sharding mapping table based on the trust degree of the device and the future capability curves of each device, and to allocate data for collaborative processing of each device using data bloodline tracking and future capability curves of each device.

[0075] The monitoring and re-sharding module is used to monitor the task execution state through the edge coordination node, to trigger dynamic re-sharding operation when detecting that the device capability deviates from the predicted value, and to verify the sharding integrity.

[0076] The data processing and updating module is used to aggregate the task execution results and return them to the requester, to record the collaborative data, and to update the trust degree and ability history library of the device.

[0077] The specific limitation of the data collaborative processing system on the data processing capability can refer to the limitation of the data collaborative processing method on the data processing capability as described above, and will not be repeated here. Each module in the above data collaborative processing system on the data processing capability can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to each module by the processor.

[0078] In an embodiment, as shown in Figure 3 An electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the data collaborative processing method on the data processing capability when executing the computer program

[0079] The specific limitation of the electronic device can refer to the limitation of the data collaborative processing method on the data processing capability as described above, and will not be repeated here.

[0080] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc. Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0081] Although the present application has been described in detail with reference to the foregoing embodiments, the technical solutions recorded in the foregoing embodiments can be modified, or some of the technical features can be replaced by equivalent features, by those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data collaborative processing method based on data processing capabilities, characterized in that, include: Collect basic capability data of the device, and construct an initial trust profile of the device based on the basic capability data. Use the initial trust profile and historical capability data to train the long short-term memory network model to obtain the device capability assessment module. After a collaborative task is input, the device capability assessment module is invoked to generate future capability curves for each device based on the device's historical capability data and current status data. The device's data processing capability is then determined based on the future capability curves and data lineage tracing. Based on the trust level of the devices, the data processing capabilities of the devices, and the future capability curves of each device, a sharding strategy is generated, and a device sharding mapping table is generated to perform data sharding. By monitoring the task execution status through the edge coordination node, if the device capability is detected to deviate from the predicted value, a dynamic resharding operation is triggered so that during the dynamic resharding process, the edge coordination node verifies the integrity of the shards and regenerates the target device sharding mapping table based on the current capability and trust level of the device. After aggregating the task execution results, the edge coordination node returns them to the requester and records the coordination data for this task. Based on the recorded coordination data, it updates the device's trust and capability history database.

2. The data collaborative processing method based on data processing capabilities according to claim 1, characterized in that, Using data lineage tracking and future capability curves for each device, data is allocated to various devices for collaborative processing, including: Send data sample packets to each device, use data lineage tracing to obtain the data processing method of each device, and obtain the data processing capability of each device; Based on the data processing capabilities of each device, the future capability curves of each device, and the trust level of each device, appropriate data is allocated to each device for collaborative data sharding.

3. The data collaborative processing method based on data processing capabilities according to claim 1, characterized in that, The trust level of the device is generated based on the historical trust level of a single device and the device's affinity, as detailed below: ； in, For the trust level of the device, 、 This is an adaptive adjustment factor for the current network environment. For the historical trust level of a single device, The user-friendliness of the equipment This is the current CPU utilization rate of the device.

4. The data collaborative processing method based on data processing capabilities according to claim 1, characterized in that, If a device capability is detected to deviate from the predicted value, a dynamic re-sharding operation is triggered, including: Mark the abnormal data slices as abnormal slices, select a high-trust device from the backup device pool to take over the task, re-encrypt the slices and distribute them, and update the mapping table.

5. The data collaborative processing method based on data processing capabilities according to claim 1, characterized in that, The collection of basic capability data includes hardware configuration: processor performance, memory capacity, storage capacity, and network connection status: bandwidth and latency.

6. The data collaborative processing method based on data processing capabilities according to claim 1, characterized in that, In the dynamic resharding operation, situations where the device capability deviates from the predicted value include sudden delays and a decrease in processing speed.

7. A data collaborative processing system based on data processing capabilities, characterized in that, include: The data collection and trust profile building module is used to collect basic device capability data during device registration and to build the device's initial trust profile. The model training module is used to deploy the long short-term memory network model and train the device capability assessment module using historical device capability data. The capability prediction module is used to call the trained device capability assessment module after inputting the collaborative task, generate the future capability curve of each device, and judge the data processing capability of the device based on the future capability curve of each device and data lineage tracing. The task allocation module is used to generate a sharding strategy and a device sharding mapping table based on the trust level of the device, the data processing capability of the device, and the future capability curve of each device, and to perform data sharding. The monitoring and resharding module is used to monitor the task execution status through edge coordination nodes. When the device capability is detected to deviate from the predicted value, a dynamic resharding operation is triggered, and the sharding integrity is verified. The data processing and update module is used to aggregate task execution results and return them to the requester, while recording collaborative data and updating the device's trust and capability history database.

8. A data collaborative processing system based on data processing capabilities according to claim 7, characterized in that, The basic capability data collected by the data acquisition and trust building module includes at least one of the following: device hardware configuration, network connection status, and historical task execution records.

9. A data collaborative processing system based on data processing capabilities according to claim 7, characterized in that, In the monitoring and resharding module, situations where the device capability deviates from the predicted value include sudden delays and a decrease in processing speed.

10. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the data collaborative processing method based on data processing capabilities according to any one of claims 1 to 6.

Images