A data sharing information system based on mass data

Abstract

The embodiment of the application relates to the computer technical field, discloses a kind of data sharing information system and method based on mass data, and the data sharing information system based on mass data of the application introduces multi-source sharing, intelligent scheduling and repair mechanism, and dynamic performance evaluation index and other innovative technologies, aims at solving many challenges that existing data sharing system faces when facing mass data, by optimizing data synchronization strategy, improving data consistency and reliability, and realizing flexible allocation of resources, the system provides strong support for efficient and reliable data sharing in mass data environment.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a data sharing information system based on massive amounts of data. Background Technology

[0002] With the rapid development of information technology, data has become an indispensable core resource in enterprise operations, scientific research, and daily life. However, in the context of massive data, achieving efficient and reliable data sharing faces a series of severe challenges. First, the explosive growth in data volume places higher demands on data storage, processing, and transmission capabilities, and traditional data sharing methods often struggle to cope with such massive amounts of data. Second, data is susceptible to interference from various factors during transmission and synchronization, such as network latency, equipment failure, and data corruption, all of which can lead to data inconsistency or loss, seriously affecting the effectiveness and reliability of data sharing.

[0003] Furthermore, most existing data sharing systems adopt a centralized architecture, where all data is processed and distributed centrally through a central server. This architecture can meet the needs when the data volume is small or network conditions are good, but when faced with massive amounts of data and complex network environments, its scalability, fault tolerance, performance bottlenecks, and sharing efficiency problems gradually become apparent. At the same time, centralized architectures also face the risk of single points of failure; if the central server fails, the entire data sharing system risks being paralyzed. Summary of the Invention

[0004] The main objective of this invention is to provide a data sharing information system based on massive amounts of data, aiming to solve the technical problems in the prior art where massive data transmission cannot meet the requirements for data sharing efficiency.

[0005] To achieve the above objectives, in a first aspect, this application provides a data sharing information system based on massive amounts of data, including a memory and a processor. The memory stores executable program code, and the processor is used to call and run the executable program code to perform the following steps: The first shared source device obtains the target shared device group based on the data synchronization command, and the target shared device group includes multiple target devices to be synchronized with data. During the process of sending shared data to the target shared device group, the data synchronization completion status of the target shared device group is monitored in real time and a second shared source device is obtained. The second shared source device is the target device that receives the shared data and completes data synchronization. Determine the consistency between the shared data of the second shared source device and the shared data of the first shared source device; In the event that the shared data of the second shared source device is inconsistent with the shared data of the first shared source device, the shared data of the second shared source device is repaired so that the second shared source device has the data sharing function. The target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and the data synchronization permission of the second target device group is assigned to the second shared source device. The first shared source device continues to complete the data sharing and synchronization of the first target device group, and the second shared source device completes the data sharing and synchronization of the second target device group after receiving the data synchronization permission.

[0006] In one possible implementation, dividing the target devices that have not completed data synchronization into a first target device group and a second target device group includes: Based on historical big data, obtain the historical transmission error rate and historical average transmission rate of target devices that have not completed data synchronization; Based on the historical transmission error rate and historical average transmission rate, a performance evaluation index is obtained for the target device that has not completed data synchronization. The performance evaluation index is used to characterize the overall data transmission performance of the target device. Based on the performance evaluation index of the target devices that have not completed data synchronization, the target devices that have not completed data synchronization are divided into a first target device group and a second target device group, wherein the sum of the performance evaluation indices of the first target device group is greater than the sum of the performance evaluation indices of the second target device group.

[0007] In one possible implementation, obtaining the performance evaluation index of the target device that has not completed data synchronization based on the historical transmission error rate and the historical average transmission rate includes: The historical transmission error rate and historical average transmission rate are input into a pre-trained performance evaluation index prediction model to obtain the performance evaluation index value. The performance evaluation index prediction model satisfies the following expression: ; Where ER is the historical transmission error rate, which is the ratio of the number of historical transmission errors to the total number of transmissions. W1 represents the historical average transmission rate index, which is the ratio of the historical average transmission rate to the device's rated transmission rate. W2 represents the weights of the historical transmission error rate and the historical average transmission rate on the performance evaluation index value.

[0008] In one possible implementation, the step of dividing the target devices that have not completed data synchronization into a first target device group and a second target device group based on the performance evaluation index of the target devices that have not completed data synchronization includes: The target devices that have not completed data synchronization are sorted from largest to smallest according to their performance evaluation index to obtain a target device queue; Starting from the tail of the target device queue, retrieve a preset proportion of target devices that have not completed data synchronization to obtain a second target device group, and then divide the remaining ones into a first target device group.

[0009] In one possible implementation, after the second sharing source device completes the data sharing synchronization of the second target device group upon receiving the data synchronization permission, the method further includes: The shared data of the second target device group is verified. If the verification of the shared data of the second target device group fails, a new request for shared data is made.

[0010] In one possible implementation, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device, after receiving the data synchronization permission, completes the data sharing synchronization of the second target device group, including: A communication link is established between the first shared source device and the second shared source device to exchange data synchronization status information about the target device group they are responsible for in real time. When the data synchronization progress of the first target device group is greater than a preset multiple of the data synchronization progress of the second target device group, the target devices in the second target device group will be transferred from the current group to the first target device group to achieve a balance in data sharing. When the data synchronization progress of the second target device group is greater than a preset multiple of the data synchronization progress of the first target device group, the target devices in the first target device group will be transferred from the current group to the second target device group to achieve a balance in data sharing.

[0011] In one possible implementation, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device, after receiving the data synchronization permission, completes the data sharing synchronization of the second target device group, including: Predict the data synchronization completion time for each target device based on a pre-trained machine learning prediction model; If the proportion of target devices whose synchronization completion time exceeds the preset duration is greater than the preset proportion value, the target devices that have completed data synchronization in the first target device group or the second target device group will be added as the third shared source device.

[0012] In one possible implementation, after the pre-trained machine learning prediction model predicts the data synchronization completion time of each target device, the method further includes: Adjust the data synchronization priority of each target device according to the data synchronization completion time of each target device, with those having shorter data synchronization completion times being adjusted to higher priority.

[0013] In one possible implementation, the repair of the shared data of the second shared source device to enable the second shared source device to have data sharing functionality includes: The shared data of the second shared source device is adjusted and repaired based on the shared data sequence adjustment trigger. A second check is performed on the shared data after the order adjustment and repair. If the order adjustment and repair fails, the shared data is rolled back and restored.

[0014] Secondly, this application provides a data sharing method based on massive amounts of data, the method comprising: The first shared source device obtains the target shared device group based on the data synchronization command, and the target shared device group includes multiple target devices to be synchronized with data. During the process of sending shared data to the target shared device group, the data synchronization completion status of the target shared device group is monitored in real time and a second shared source device is obtained. The second shared source device is the target device that receives the shared data and completes data synchronization. Determine the consistency between the shared data of the second shared source device and the shared data of the first shared source device; In the event that the shared data of the second shared source device is inconsistent with the shared data of the first shared source device, the shared data of the second shared source device is repaired so that the second shared source device has the data sharing function. The target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and the data synchronization permission of the second target device group is assigned to the second shared source device. The first shared source device continues to complete the data sharing and synchronization of the first target device group, and the second shared source device completes the data sharing and synchronization of the second target device group after receiving the data synchronization permission.

[0015] Unlike existing technologies, the data sharing information system and method based on massive data provided in this application first obtains a target sharing device group based on a data synchronization command. During the process of sending shared data to the target sharing device group, the data synchronization completion status of the target sharing device group is monitored in real time, and a second sharing source device is obtained. Then, the consistency between the shared data of the second sharing source device and the shared data of the first sharing source device is determined. If the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, the shared data of the second sharing source device is repaired to enable the second sharing source device to have data sharing functionality. Next, the target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and the data synchronization permission of the second target device group is allocated to the second sharing source device. Finally, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization permission. This system aims to address the numerous challenges faced by existing data sharing systems when dealing with massive amounts of data by introducing innovative technologies such as multi-source sharing, intelligent scheduling and repair mechanisms, and dynamic performance evaluation indices. By optimizing data synchronization strategies, improving data consistency and reliability, and enabling flexible resource allocation, this system provides strong support for efficient and reliable data sharing in massive data environments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a data sharing method based on massive amounts of data in some embodiments of this application; Figure 2 This is a flowchart illustrating a data sharing method based on massive amounts of data in other embodiments of this application; Figure 3 This is a schematic diagram of the hardware structure of a data sharing information system based on massive data in some embodiments of this application.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0021] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the term "and / or" throughout the text includes three solutions; taking A and / or B as an example, it includes technical solution A, technical solution B, and a technical solution that simultaneously satisfies A and B. Furthermore, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of a person skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0022] With the rapid development of information technology, data has become an indispensable core resource in enterprise operations, scientific research, and daily life. However, in the context of massive data, achieving efficient and reliable data sharing faces a series of severe challenges. First, the explosive growth in data volume places higher demands on data storage, processing, and transmission capabilities, and traditional data sharing methods often struggle to cope with such massive amounts of data. Second, data is susceptible to interference from various factors during transmission and synchronization, such as network latency, equipment failure, and data corruption, all of which can lead to data inconsistency or loss, seriously affecting the effectiveness and reliability of data sharing.

[0023] Furthermore, most existing data sharing systems adopt a centralized architecture, where all data is processed and distributed centrally through a central server. This architecture can meet the needs when the data volume is small or network conditions are good, but when faced with massive amounts of data and complex network environments, its scalability, fault tolerance, performance bottlenecks, and sharing efficiency problems gradually become apparent. At the same time, centralized architectures also face the risk of single points of failure; if the central server fails, the entire data sharing system risks being paralyzed.

[0024] To address the aforementioned problems, this application proposes a data sharing method based on massive amounts of data. This method is applied to a first sharing source device, such as... Figure 1-2 As shown, the following will mainly describe the specific steps of a data sharing method based on massive amounts of data. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here. Please refer to the appendix. Figure 1 The method includes the following steps: S100, The first shared source device obtains the target shared device group based on the data synchronization command, the target shared device group including multiple target devices to be synchronized with data; The first sharing source device can be a server, a data center, or any device capable of initiating data sharing. This application uses a physical device capable of initiating data sharing as an example for illustration.

[0025] The first sharing source device obtains the target sharing device group according to the data synchronization command. The target sharing device group can be target devices that need to synchronize data with the first sharing source device on the same local area network, that is, devices that need to receive shared data to realize their own functions. For example, receiving OTA upgrade packages to upgrade firmware.

[0026] S200: During the process of sending shared data to the target shared device group, the data synchronization completion status of the target shared device group is monitored in real time and a second shared source device is obtained. The second shared source device is the target device that receives the shared data and completes data synchronization. Specifically, during the process of the first sharing source device sending shared data to the target sharing device group, the system monitors the synchronization progress and completion status of each target device in real time. When a target device receives and successfully synchronizes the shared data, it is marked as the second sharing source device (i.e., a device that already has complete synchronized data, in addition to the initial data source). In this way, potential problems during the synchronization process can be detected and resolved in a timely manner, while identifying devices that have completed synchronization so that they can participate in subsequent data synchronization tasks.

[0027] S300: Determine the consistency between the shared data of the second shared source device and the shared data of the first shared source device; Specifically, the system verifies the shared data of the second shared source device to ensure that it is completely consistent with the original shared data of the first shared source device, thereby ensuring the integrity and accuracy of the data and preventing data inconsistencies caused by network transmission errors, equipment failures, etc., from affecting subsequent data synchronization tasks.

[0028] It should be noted that data consistency can include the consistency of data content and the consistency of the order of data modules.

[0029] S400: If the shared data of the second shared source device is inconsistent with the shared data of the first shared source device, the shared data of the second shared source device is repaired so that the second shared source device has the data sharing function. It is understandable that when the shared data of the second shared source device is found to be inconsistent with that of the first shared source device, it indicates that the second shared source device cannot perform normal data sharing. In this case, appropriate repair measures will be taken, such as retransmitting the missing or damaged data portions, until the shared data of the second shared source device is completely restored to consistency. Alternatively, when the error rate of the data is low, its own repair algorithm will be used to repair the data itself.

[0030] In one embodiment, the step of repairing the shared data of the second shared source device to enable the second shared source device to have data sharing functionality includes: The shared data of the second shared source device is adjusted and repaired based on the shared data sequence adjustment trigger. A second check is performed on the shared data after the order adjustment and repair. If the order adjustment and repair fails, the shared data is rolled back and restored.

[0031] Specifically, one or more triggers are pre-designed to monitor specific errors or abnormal conditions during data synchronization, such as incorrect data block reception order or discontinuous timestamps. When the trigger condition is met, the trigger automatically activates, initiating the order adjustment and repair process. The repaired data is then written back to the storage medium of the second shared source device, replacing the original out-of-order data. If the order adjustment and repair fails during secondary detection (i.e., the repaired data still has problems or is inconsistent with the original data), a shared data rollback recovery is required. Relevant information about the rollback operation, including the reason, time, and personnel involved, is recorded. Relevant personnel (such as system administrators and data owners) are notified of the rollback situation so they can understand and take appropriate follow-up measures. Through this series of steps, order problems that occur during data synchronization can be effectively handled while ensuring data consistency and reliability.

[0032] S500: Divide the target devices that have not completed data synchronization into a first target device group and a second target device group, and assign the data synchronization permission of the second target device group to the second shared source device; Specifically, during the data sharing process, target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and data synchronization permissions for the second target device group are assigned to the second sharing source device. It can be understood that the first target device group and the second target device group can respectively achieve the data sharing and synchronization task through the first sharing source device and the second sharing source device. The division of target devices that have not completed data synchronization can be random or based on device performance to achieve reasonable matching of target devices.

[0033] In one embodiment, the step of dividing the target devices that have not completed data synchronization into a first target device group and a second target device group includes: Based on historical big data, obtain the historical transmission error rate and historical average transmission rate of target devices that have not completed data synchronization; Based on the historical transmission error rate and historical average transmission rate, a performance evaluation index is obtained for the target device that has not completed data synchronization. The performance evaluation index is used to characterize the overall data transmission performance of the target device. Based on the performance evaluation index of the target devices that have not completed data synchronization, the target devices that have not completed data synchronization are divided into a first target device group and a second target device group, wherein the sum of the performance evaluation indices of the first target device group is greater than the sum of the performance evaluation indices of the second target device group.

[0034] Specifically, the inventors pre-constructed a performance evaluation index prediction model. In constructing this model, they considered the weights of the historical transmission error rate and historical average transmission rate of the target device on the performance evaluation index value. Furthermore, considering the negative correlation between transmission error rate and device performance, and the positive correlation between transmission rate and device performance, the performance evaluation index prediction model was obtained based on a large amount of training data and after data fitting. The expression for the performance evaluation index prediction model is as follows: PEI is the performance evaluation index estimate, and ER is the historical transmission error rate, which is the ratio of the number of historical transmission errors to the total number of transmissions. The historical average transmission rate index is the ratio of the historical average transmission rate to the device's rated transmission rate. W1 and W2 are the weights of the historical transmission error rate and historical average transmission rate on the performance evaluation index value. After obtaining the historical transmission error rate and historical average transmission rate of the target devices that have not completed data synchronization based on historical big data, the historical transmission error rate and historical average transmission rate are input into the performance evaluation index prediction model to obtain the performance evaluation index. Then, based on the performance evaluation index of the target devices that have not completed data synchronization, the target devices that have not completed data synchronization are divided into a first target device group and a second target device group.

[0035] It should be noted that, for the convenience of estimating performance evaluation indices, the inventors considered the historical transmission error rate and historical average transmission rate of the target device to be of equal importance. All values ​​were assigned a value of 0.5. Furthermore, in the performance evaluation index prediction model expression, the applicant introduced an exponential function for the transmission rate component during the data fitting process. Thus, as the transmission rate increases, its contribution to PEI grows exponentially; this reflects that in practical applications, high-speed transmission often has a higher value than low-speed transmission.

[0036] In one embodiment, the step of dividing the target devices that have not completed data synchronization into a first target device group and a second target device group based on the performance evaluation index of the target devices that have not completed data synchronization includes: The target devices that have not completed data synchronization are sorted from largest to smallest according to their performance evaluation index to obtain a target device queue; Starting from the tail of the target device queue, retrieve a preset proportion of target devices that have not completed data synchronization to obtain a second target device group, and then divide the remaining ones into a first target device group.

[0037] Specifically, a target device queue is obtained by sorting the target devices according to their overall data transmission performance. A first target device group is obtained by taking a portion of the data from the head of the target device queue, and a second target device group is obtained by taking a portion of the data from the tail of the target device queue. For example, if there are 10 target devices in the target device queue, the top 5 in terms of overall data transmission performance can be selected as the first target device group, and the bottom 5 can be selected as the second target device group. Since the second shared source device is a shared source device obtained after data repair, its performance is relatively lower than that of the first shared source device. Therefore, this application uses the first shared source device to perform data sharing and synchronization for high-performance target devices, and uses the second shared source device to perform data sharing and synchronization for low-performance target devices. By matching the high-performance first shared source device with the high-performance target devices, the high-speed processing capabilities of both can be fully utilized to achieve high-speed data transmission and synchronization. Meanwhile, equipping low-performance target devices with suitable shared source devices can ensure the stability of data transmission and avoid inefficiency or errors caused by performance mismatch. Furthermore, this application helps to allocate resources reasonably, avoid resource waste caused by high-performance devices processing data from low-performance target devices, and also ensure that low-performance target devices can obtain sufficient data support.

[0038] Furthermore, since the second shared source device is a data-repaired device with lower performance, the effectiveness of using it for data sharing and synchronization needs further verification. This means that the shared data of the second target device group can be verified. If the verification fails, a new data sharing request should be made. In other words, after using the second shared source device to synchronize data with the second target device group, the shared data of the second target device group needs to be verified to ensure successful data sharing and synchronization.

[0039] S600, the first shared source device continues to complete the data sharing synchronization of the first target device group, and the second shared source device completes the data sharing synchronization of the second target device group after receiving the data synchronization permission.

[0040] It is understandable that when the target devices that have not completed data synchronization are randomly divided in step 500, the different states of each target device lead to significant differences in the data sharing efficiency between the first target device group and the second target device group. Therefore, during the data sharing synchronization process, data sharing balance can be achieved based on the synchronization status of the first sharing source device and the second sharing source device. In one embodiment, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization permission, including: S610. Establish a communication link between the first shared source device and the second shared source device to exchange data synchronization status information about the target device group they are responsible for in real time. S620. When the data synchronization progress of the first target device group is greater than a preset multiple of the data synchronization progress of the second target device group, the target devices in the second target device group are transferred from the current group to the first target device group to achieve a balance in data sharing. S630. When the data synchronization progress of the second target device group is greater than a preset multiple of the data synchronization progress of the first target device group, the target devices in the first target device group are transferred from the current group to the second target device group to achieve a balance in data sharing.

[0041] Specifically, by establishing a communication link between the first and second shared source devices to exchange data synchronization status information about the target device groups they are responsible for in real time, the overall data synchronization efficiency and accuracy are optimized. When any shared source device detects a data synchronization delay or error in its responsible target device group, it can notify the other party through the communication link and negotiate adjustments to the data synchronization strategy to restore data consistency across all target devices as quickly as possible. For example, if the data synchronization progress of a certain target device group is significantly faster or slower than other device groups, the system can automatically or according to user instructions transfer the device from the current group to another group to more effectively utilize the processing power of each shared source device and achieve load balancing for data synchronization. The synchronization progress can be determined by the number of devices that complete synchronization at the same time.

[0042] In other embodiments, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization permission, including: Predict the data synchronization completion time for each target device based on a pre-trained machine learning prediction model; If the proportion of target devices whose synchronization completion time exceeds the preset duration is greater than the preset proportion value, the target devices that have completed data synchronization in the first target device group or the second target device group will be added as the third shared source device.

[0043] Specifically, the system first predicts the data synchronization completion time for each target device based on a pre-trained machine learning prediction model. When the system detects that the proportion of target devices with synchronization completion times exceeding a preset threshold is greater than a preset percentage, it indicates that data synchronization for most target devices is slow. At this point, the system automatically triggers a resource optimization mechanism, selecting a high-performance, resource-rich target device from either the first or second group of target devices that have already completed data synchronization, and adding it as a third shared source device. Specifically, the newly added third shared source device will assist the original first and second shared source devices in jointly undertaking the data synchronization task. The system intelligently allocates data synchronization tasks to the third shared source device based on the current load, performance parameters, and remaining available resources of each shared source device to balance the overall load, reduce data synchronization latency, and further improve the efficiency and reliability of the entire data synchronization system.

[0044] In other embodiments, the data synchronization priority of each target device can be adjusted according to its data synchronization completion time, with those having shorter completion times being given higher priority. In other words, target devices that are easier to synchronize are prioritized for data sharing synchronization. By prioritizing these easier-to-synchronize target devices, the number of devices in the system that have not yet completed synchronization can be rapidly reduced, thereby accelerating the entire data sharing synchronization process.

[0045] Based on this, the data sharing method based on massive data in this application first obtains a target sharing device group based on a data synchronization command. During the process of sending shared data to the target sharing device group, the data synchronization completion status of the target sharing device group is monitored in real time, and a second sharing source device is obtained. Then, the consistency between the shared data of the second sharing source device and the shared data of the first sharing source device is determined. If the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, the shared data of the second sharing source device is repaired so that the second sharing source device has the data sharing function. Then, the target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and the data synchronization permission of the second target device group is assigned to the second sharing source device. Finally, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization permission. This system aims to address the numerous challenges faced by existing data sharing systems when dealing with massive amounts of data by introducing innovative technologies such as multi-source sharing, intelligent scheduling and repair mechanisms, and dynamic performance evaluation indices. By optimizing data synchronization strategies, improving data consistency and reliability, and enabling flexible resource allocation, this system provides strong support for efficient and reliable data sharing in massive data environments.

[0046] This application also provides a data sharing information system based on massive amounts of data. Please refer to the appendix. Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of a data sharing information system based on massive data provided in some embodiments of this application; it includes a first sharing source device, a memory 110, and a processor 120, wherein the memory 110 is used to store program code, and the processor 120 is used to call the program code to execute the method described above.

[0047] The processor 120 provides computing and control capabilities to control the data sharing information system based on massive data to perform corresponding tasks. For example, it controls the data sharing information system based on massive data to perform the data sharing method based on massive data in any of the above method embodiments. The method includes: a first sharing source device acquiring a target sharing device group based on a data synchronization instruction, the target sharing device group including multiple target devices to be synchronized; monitoring the data synchronization completion status of the target sharing device group in real time during the process of sending shared data to the target sharing device group and obtaining a second sharing source device, the second sharing source device being the target device that receives the shared data and completes data synchronization; determining the second... The shared data of the shared source device is consistent with the shared data of the first shared source device; if the shared data of the second shared source device is inconsistent with the shared data of the first shared source device, the shared data of the second shared source device is repaired so that the second shared source device has the data sharing function; the target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and the data synchronization permission of the second target device group is assigned to the second shared source device; the first shared source device continues to complete the data sharing synchronization of the first target device group, and the second shared source device completes the data sharing synchronization of the second target device group after receiving the data synchronization permission.

[0048] The processor 120 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0049] The memory 110, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the data sharing method based on massive data in the embodiments of this application. The processor 120 can implement the data sharing method based on massive data in any of the above method embodiments by running the non-transitory software programs, instructions, and modules stored in the memory 110.

[0050] Specifically, memory 110 may include volatile memory (VM), such as random access memory (RAM); memory 110 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or other non-transitory solid-state storage devices; memory 110 may also include combinations of the above types of memory.

[0051] In summary, the data sharing information system based on massive data in this application adopts the technical solution of any of the above-mentioned data sharing method embodiments based on massive data. Therefore, it has at least the beneficial effects brought by the technical solutions of the above embodiments, which will not be elaborated here.

[0052] This application also provides a computer-readable storage medium, such as a memory including program code, which can be executed by a processor to complete the data sharing method based on massive data described in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0053] This application also provides a computer program product comprising one or more lines of program code stored in a computer-readable storage medium. A processor of a data sharing information system based on massive data reads the program code from the computer-readable storage medium and executes the program code to complete the steps of the data sharing method based on massive data provided in the above embodiments.

[0054] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program or program code related to hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0055] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0056] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0057] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A data sharing information system based on massive amounts of data, characterized in that, The device includes a first shared source device, a memory, and a processor. The memory stores executable program code, and the processor is used to call and run the executable program code to perform the following steps: The first shared source device obtains the target shared device group based on the data synchronization command. The target shared device group consists of target devices that need to be synchronized with the first shared source device in the same local area network. The target shared device group includes multiple target devices to be synchronized with the data. During the process of sending shared data to the target shared device group, the data synchronization completion status of the target shared device group is monitored in real time and a second shared source device is obtained. The second shared source device is the target device that receives the shared data and completes data synchronization. Determine the consistency between the shared data of the second shared source device and the shared data of the first shared source device, wherein the data consistency includes the consistency of data content and the consistency of data module order; In the event that the shared data of the second shared source device is inconsistent with the shared data of the first shared source device, the shared data of the second shared source device is repaired so that the second shared source device has the data sharing function. The target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and the data synchronization permission of the second target device group is assigned to the second shared source device. The first shared source device continues to complete the data sharing and synchronization of the first target device group, and the second shared source device completes the data sharing and synchronization of the second target device group after receiving the data synchronization permission; The step of repairing the shared data of the second shared source device to enable the second shared source device to have data sharing functionality includes: The shared data of the second shared source device is adjusted and repaired based on the shared data sequence adjustment trigger. A second check is performed on the shared data after the order adjustment and repair. If the order adjustment and repair fails, the shared data is rolled back and restored.

2. The data sharing information system based on massive data as described in claim 1, characterized in that, The step of dividing the target devices that have not completed data synchronization into a first target device group and a second target device group includes: Based on historical big data, obtain the historical transmission error rate and historical average transmission rate of target devices that have not completed data synchronization; Based on the historical transmission error rate and historical average transmission rate, a performance evaluation index is obtained for the target device that has not completed data synchronization. The performance evaluation index is used to characterize the overall data transmission performance of the target device. Based on the performance evaluation index of the target devices that have not completed data synchronization, the target devices that have not completed data synchronization are divided into a first target device group and a second target device group, wherein the sum of the performance evaluation indices of the first target device group is greater than the sum of the performance evaluation indices of the second target device group.

3. The data sharing information system based on massive data as described in claim 2, characterized in that, The performance evaluation index of the target device that has not completed data synchronization, obtained based on the historical transmission error rate and historical average transmission rate, includes: The historical transmission error rate and historical average transmission rate are input into a pre-trained performance evaluation index prediction model to obtain the performance evaluation index value. The performance evaluation index prediction model satisfies the following expression: Where ER is the historical transmission error rate, which is the ratio of the number of historical transmission errors to the total number of transmissions. W1 represents the historical average transmission rate index, which is the ratio of the historical average transmission rate to the device's rated transmission rate. W2 represents the weights of the historical transmission error rate and the historical average transmission rate on the performance evaluation index value.

4. The data sharing information system based on massive data as described in claim 3, characterized in that, The process of dividing the target devices that have not completed data synchronization into a first target device group and a second target device group based on the performance evaluation index of the target devices that have not completed data synchronization includes: The target devices that have not completed data synchronization are sorted from largest to smallest according to their performance evaluation index to obtain a target device queue; Starting from the tail of the target device queue, retrieve a preset proportion of target devices that have not completed data synchronization to obtain a second target device group, and then divide the remaining ones into a first target device group.

5. The data sharing information system based on massive data as described in claim 4, characterized in that, After the second sharing source device receives the data synchronization permission and completes the data sharing synchronization of the second target device group, it further includes: The shared data of the second target device group is verified. If the verification of the shared data of the second target device group fails, a new shared data request is made.

6. The data sharing information system based on massive data as described in claim 1, characterized in that, The first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device, after receiving the data synchronization permission, completes the data sharing synchronization of the second target device group, including: A communication link is established between the first shared source device and the second shared source device to exchange data synchronization status information about the target device group they are responsible for in real time. When the data synchronization progress of the first target device group is greater than a preset multiple of the data synchronization progress of the second target device group, the target devices in the second target device group will be transferred from the current group to the first target device group to achieve a balance in data sharing. When the data synchronization progress of the second target device group is greater than a preset multiple of the data synchronization progress of the first target device group, the target devices in the first target device group will be transferred from the current group to the second target device group to achieve a balance in data sharing.

7. The data sharing information system based on massive data as described in claim 1, characterized in that, The first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device, after receiving the data synchronization permission, completes the data sharing synchronization of the second target device group, including: Predict the data synchronization completion time for each target device based on a pre-trained machine learning prediction model; If the proportion of target devices whose synchronization completion time exceeds the preset duration is greater than the preset proportion value, the target devices that have completed data synchronization in the first target device group or the second target device group will be added as the third shared source device.

8. The data sharing information system based on massive data as described in claim 7, characterized in that, After the pre-trained machine learning prediction model predicts the data synchronization completion time of each target device, the method further includes: Adjust the data synchronization priority of each target device according to the data synchronization completion time of each target device, with those having shorter data synchronization completion times being adjusted to higher priority.

9. The data sharing information system based on massive data as described in claim 1, characterized in that, Triggers are used to listen for specific errors or abnormal conditions during data synchronization. When the triggering condition is met, the trigger is automatically activated, the order adjustment and repair process is started, and then the repaired data is written back to the storage medium of the second shared source device to replace the original out-of-order data.

10. A data sharing method based on massive amounts of data, characterized in that, The method includes: The first shared source device obtains the target shared device group based on the data synchronization command. The target shared device group consists of target devices that need to be synchronized with the first shared source device in the same local area network. The target shared device group includes multiple target devices to be synchronized with the data. During the process of sending shared data to the target shared device group, the data synchronization completion status of the target shared device group is monitored in real time and a second shared source device is obtained. The second shared source device is the target device that receives the shared data and completes data synchronization. Determine the consistency between the shared data of the second shared source device and the shared data of the first shared source device, wherein the data consistency includes the consistency of data content and the consistency of data module order; In the event that the shared data of the second shared source device is inconsistent with the shared data of the first shared source device, the shared data of the second shared source device is repaired so that the second shared source device has the data sharing function. The target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and the data synchronization permission of the second target device group is assigned to the second shared source device. The first shared source device continues to complete the data sharing and synchronization of the first target device group, and the second shared source device completes the data sharing and synchronization of the second target device group after receiving the data synchronization permission; The step of repairing the shared data of the second shared source device to enable the second shared source device to have data sharing functionality includes: The shared data of the second shared source device is adjusted and repaired based on the shared data sequence adjustment trigger. A second check is performed on the shared data after the order adjustment and repair. If the order adjustment and repair fails, the shared data is rolled back and restored.

Images