Data sharing information system based on mass data

Abstract

The embodiment of the invention relates to the technical field of computers, and discloses a data sharing information system and method based on mass data. The data sharing information system based on mass data introduces innovative technologies such as multi-source sharing, intelligent scheduling and repairing mechanisms and dynamic performance evaluation indexes; the system aims to solve many challenges faced by an existing data sharing system when facing mass data, and provides powerful support for efficient and reliable data sharing in a mass data environment by optimizing a data synchronization strategy, improving data consistency and reliability and realizing flexible allocation of resources.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data sharing information system based on mass data. Background Art

[0002] With the rapid development of information technology, data has become an indispensable core resource in business operations, scientific research and daily life. However, in the environment of massive data, achieving efficient and reliable data sharing faces a series of severe challenges. First, the explosive growth of data volume has put forward higher requirements on data storage, processing and transmission capabilities. Traditional data sharing methods are often unable to cope with such a huge amount of data. Secondly, data is easily interfered by various factors during transmission and synchronization, such as network delays, equipment failures, data corruption, etc., which may lead to data inconsistency or loss, seriously affecting the effectiveness and reliability of data sharing.

[0003] In addition, most existing data sharing systems adopt a centralized architecture, that is, all data is centrally processed and distributed through a central server. This architecture can still meet the needs when the data volume is small or the network conditions are good, but when faced with massive data and complex network environments, its scalability, fault tolerance, performance bottlenecks, and sharing efficiency problems gradually emerge. At the same time, the centralized architecture also faces the risk of single point failure. Once the central server fails, the entire data sharing system will face the risk of paralysis. Summary of the invention

[0004] The main purpose of the present invention is to provide a data sharing information system based on massive data, aiming to solve the technical problem that the massive data transmission in the prior art cannot meet the requirements of data sharing efficiency and the like.

[0005] To achieve the above objectives, in a first aspect, an embodiment of the present application provides a data sharing information system based on massive data, including a memory and a processor, wherein the memory stores executable program code, and the processor is used to implement the following steps when calling and running the executable program code: The first sharing source device acquires a target sharing device group based on a data synchronization instruction, wherein the target sharing device group includes a plurality of target devices to be synchronized with data; In the process of sending shared data to the target shared device group, real-time monitoring of the data synchronization completion status of the target shared device group is performed and a second shared source device is obtained, where the second shared source device is a target device that receives the shared data and completes data synchronization; determining consistency between the shared data of the second sharing source device and the shared data of the first sharing source device; When the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, repairing the shared data of the second sharing source device so that the second sharing source device has a data sharing function; Dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group, and allocating the data synchronization authority of the second target device group to the second sharing source device; The first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority.

[0006] In a possible implementation, dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group includes: Based on historical big data, the historical transmission error rate and historical average transmission rate of the target device that has not completed data synchronization are obtained; Obtaining a performance evaluation index of the target device for which data synchronization has not been completed according to the historical transmission error rate and the historical average transmission rate, wherein the performance evaluation index is used to characterize the overall performance of data transmission of the target device; The target devices for which data synchronization has not been completed are divided into a first target device group and a second target device group according to the performance evaluation indexes of the target devices for which data synchronization has not been completed, wherein the sum of the performance evaluation indexes of the first target device group is greater than the sum of the performance evaluation indexes of the second target device group.

[0007] In a possible implementation, obtaining the performance evaluation index of the target device for which data synchronization has not been completed according to the historical transmission error rate and the historical average transmission rate includes: The historical transmission error rate and the historical average transmission rate are input into a pre-trained performance evaluation index prediction model to obtain a performance evaluation index value, and the performance evaluation index prediction model satisfies the following expression: ; Among them, ER is the historical transmission error rate, which is the ratio of the number of historical transmission errors to the total number of transmissions. is the historical average transmission rate index, is the ratio of the historical average transmission rate to the device’s rated transmission rate, and W 1 , W 2 is the influence weight of the historical transmission error rate and the historical average transmission rate on the performance evaluation index value.

[0008] In a possible implementation, dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group according to the performance evaluation index of the target devices for which data synchronization has not been completed includes: Sorting the target devices that have not completed data synchronization from large to small according to performance evaluation indexes to obtain a target device queue; A preset number of target devices with uncompleted data synchronization are acquired in reverse order from the tail of the target device queue to obtain a second target device group, and the rest are divided into a first target device group.

[0009] In a possible implementation, after the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization permission, the method further includes: The shared data of the second target device group is verified, and if the shared data of the second target device group fails to be verified, a shared data request is made again.

[0010] In a possible implementation, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority, including: Establishing a communication link between the first sharing source device and the second sharing source device to exchange data synchronization status information about the target device groups that each device is responsible for in real time; When the data synchronization progress of the first target device group is greater than a preset multiple of the data synchronization progress of the second target device group, the target devices in the second target device group are transferred from the current group to the first target device group to achieve a balance in data sharing; When the data synchronization progress of the second target device group is greater than a preset multiple of the data synchronization progress of the first target device group, the target devices in the first target device group are transferred from the current group to the second target device group to achieve a balance in data sharing.

[0011] In a possible implementation, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority, including: Predict the data synchronization completion time of each target device based on a pre-trained machine learning prediction model; When the proportion of target devices whose synchronization completion time is greater than the preset time is greater than the preset proportion value, the target devices that have completed data synchronization in the first target device group or the second target device group are added as third sharing source devices.

[0012] In a possible implementation, after predicting the data synchronization completion time of each target device based on the pre-trained machine learning prediction model, the method further includes: The data synchronization priority of each target device is adjusted according to the data synchronization completion time of each target device, wherein the data synchronization completion time is adjusted to a high priority.

[0013] In a possible implementation manner, repairing the shared data of the second sharing source device so that the second sharing source device has a data sharing function includes: Performing sequence adjustment and repair on the shared data of the second sharing source device based on a shared data sequence adjustment trigger; A secondary check is performed on the shared data after the sequence adjustment and repair, and a rollback recovery of the shared data is performed when the sequence adjustment and repair fails.

[0014] In a second aspect, an embodiment of the present application provides a data sharing method based on massive data, the method comprising: The first sharing source device acquires a target sharing device group based on a data synchronization instruction, wherein the target sharing device group includes a plurality of target devices to be synchronized with data; In the process of sending shared data to the target shared device group, real-time monitoring of the data synchronization completion status of the target shared device group is performed and a second shared source device is obtained, where the second shared source device is a target device that receives the shared data and completes data synchronization; determining consistency between the shared data of the second sharing source device and the shared data of the first sharing source device; When the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, repairing the shared data of the second sharing source device so that the second sharing source device has a data sharing function; Dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group, and allocating the data synchronization authority of the second target device group to the second sharing source device; The first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority.

[0015] Different from the prior art, the data sharing information system and method based on massive data provided by the embodiment of the present application first obtains the target sharing device group based on the data synchronization instruction, monitors the data synchronization completion status of the target sharing device group in real time and obtains the second sharing source device during the process of sending shared data to the target sharing device group, and then determines the consistency between the shared data of the second sharing source device and the shared data of the first sharing source device; when the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, repairs the shared data of the second sharing source device so that the second sharing source device has the data sharing function; then divides the target devices that have not completed data synchronization into the first target device group and the second target device group, and allocates the data synchronization authority of the second target device group to the second sharing source device; finally, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority. The system aims to solve the many challenges faced by existing data sharing systems when faced with massive data by introducing innovative technologies such as multi-source sharing, intelligent scheduling and repair mechanisms, and dynamic performance evaluation indexes. By optimizing data synchronization strategies, improving data consistency and reliability, and achieving flexible allocation of resources, the system provides strong support for efficient and reliable data sharing in massive data environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of a flow chart of a data sharing method based on massive data in some embodiments of the present application; Figure 2 A schematic diagram of a flow chart of a data sharing method based on massive data in some other embodiments of the present application; Figure 3 This is a schematic diagram of the hardware structure of a data sharing information system based on massive data in some embodiments of the present application.

[0018] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0021] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, "and / or" in the full text includes three solutions. Taking A and / or B as an example, it includes technical solution A, technical solution B, and technical solution that satisfies both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0022] With the rapid development of information technology, data has become an indispensable core resource in business operations, scientific research and daily life. However, in the environment of massive data, achieving efficient and reliable data sharing faces a series of severe challenges. First, the explosive growth of data volume has put forward higher requirements on data storage, processing and transmission capabilities. Traditional data sharing methods are often unable to cope with such a huge amount of data. Secondly, data is easily interfered by various factors during transmission and synchronization, such as network delays, equipment failures, data corruption, etc., which may lead to data inconsistency or loss, seriously affecting the effectiveness and reliability of data sharing.

[0023] In addition, most existing data sharing systems adopt a centralized architecture, that is, all data is centrally processed and distributed through a central server. This architecture can still meet the needs when the data volume is small or the network conditions are good, but when faced with massive data and complex network environments, its scalability, fault tolerance, performance bottlenecks, and sharing efficiency problems gradually emerge. At the same time, the centralized architecture also faces the risk of single point failure. Once the central server fails, the entire data sharing system will face the risk of paralysis.

[0024] In view of the above problems, the present application proposes a data sharing method based on massive data, which is applied to a first sharing source device, such as Figure 1-2 As shown, the following will mainly describe the specific steps of the data sharing method based on massive data. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order from here. Figure 1 , the method comprises the following steps: S100, a first sharing source device acquires a target sharing device group based on a data synchronization instruction, wherein the target sharing device group includes a plurality of target devices to be synchronized with data; The first sharing source device may be a server, a data center, or any device capable of initiating data sharing. This application is described by taking a physical device capable of initiating data sharing as an example.

[0025] The first sharing source device obtains the target sharing device group according to the data synchronization instruction. The target sharing device group may be a target device in the same local area network as the first sharing source device that needs to synchronize data, that is, needs to receive shared data to realize its own functions. For example, receiving an OTA upgrade package for firmware upgrade.

[0026] S200, in the process of sending shared data to the target shared device group, real-time monitoring of the data synchronization completion status of the target shared device group and obtaining a second shared source device, where the second shared source device is a target device that receives the shared data and completes data synchronization; Specifically, when the first sharing source device sends shared data to the target sharing device group, the system monitors the synchronization progress and completion status of each target device in real time; when a target device receives and successfully synchronizes the shared data, it is marked as the second sharing source device (i.e., a device that already has complete synchronized data in addition to the initial data source). In this way, problems that may arise during the synchronization process can be discovered and resolved in a timely manner, and devices that have completed synchronization can be identified so that they can participate in subsequent data synchronization tasks.

[0027] S300, determining the consistency between the shared data of the second sharing source device and the shared data of the first sharing source device; Specifically, the system verifies the shared data of the second shared source device to ensure that it is completely consistent with the original shared data of the first shared source device, ensures the integrity and accuracy of the data, and prevents data inconsistencies caused by network transmission errors, equipment failures, etc., which affect subsequent data synchronization tasks.

[0028] It should be noted that data consistency may include consistency of data content and consistency of data module order, etc.

[0029] S400: When the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, repair the shared data of the second sharing source device so that the second sharing source device has a data sharing function; It can be understood that when it is found that the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, it means that the second sharing source device cannot realize the normal data sharing function, and appropriate repair measures will be taken at this time, such as resending the missing or damaged data part, until the shared data of the second sharing source device is completely restored to consistency. Or, when the error degree of the data is low, the data is self-repaired by using its own repair algorithm.

[0030] In one embodiment, the step of repairing the shared data of the second sharing source device so that the second sharing source device has a data sharing function includes: Performing sequence adjustment and repair on the shared data of the second sharing source device based on a shared data sequence adjustment trigger; A secondary check is performed on the shared data after the sequence adjustment and repair, and a rollback recovery of the shared data is performed when the sequence adjustment and repair fails.

[0031] Specifically, one or more triggers are pre-designed, and these triggers monitor specific errors or abnormal conditions during the data synchronization process, such as errors in the order of receiving data blocks, discontinuous timestamps, etc.; when the trigger conditions are met, the trigger is automatically activated, the order adjustment and repair process is started, and then the repaired data is written back to the storage medium of the second shared source device to replace the original disordered data. If the order adjustment and repair fails in the secondary detection (that is, the repaired data still has problems or is inconsistent with the original data), it is necessary to roll back the shared data and record the relevant information of the rollback operation, including the reason, time, operator, etc. of the rollback, and notify relevant personnel (such as system administrators, data owners, etc.) about the rollback so that they can understand and take corresponding follow-up measures. Through this series of steps, the order problems that arise in the data synchronization process can be effectively handled while ensuring data consistency and reliability.

[0032] S500, dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group, and allocating the data synchronization authority of the second target device group to the second sharing source device; Specifically, during the data sharing process, the target devices that have not completed data synchronization are divided into a first target device group and a second target device group, and the data synchronization authority of the second target device group is allocated to the second shared source device. It can be understood that the first target device group and the second target device group can respectively implement the data sharing synchronization task through the first shared source device and the second shared source device. The target devices that have not completed data synchronization can be divided randomly or according to the performance of the device to reasonably match the target devices.

[0033] In one embodiment, the step of dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group includes: Based on historical big data, the historical transmission error rate and historical average transmission rate of the target device that has not completed data synchronization are obtained; Obtaining a performance evaluation index of the target device for which data synchronization has not been completed according to the historical transmission error rate and the historical average transmission rate, wherein the performance evaluation index is used to characterize the overall performance of data transmission of the target device; The target devices for which data synchronization has not been completed are divided into a first target device group and a second target device group according to the performance evaluation indexes of the target devices for which data synchronization has not been completed, wherein the sum of the performance evaluation indexes of the first target device group is greater than the sum of the performance evaluation indexes of the second target device group.

[0034] Specifically, the inventor pre-constructed a performance evaluation index prediction model. When constructing the performance evaluation index prediction model, the influence weights of the historical transmission error rate and the historical average transmission rate of the target device on the performance evaluation index value were considered respectively. In addition, considering that the transmission error rate is negatively correlated with the performance of the device, and the transmission rate is positively correlated with the performance of the device, the performance evaluation index prediction model was obtained after fitting the data based on a large amount of training data. The expression of the performance evaluation index prediction model is: , PEI is the estimated value of the performance evaluation index, ER is the historical transmission error rate, which is the ratio of the number of historical transmission errors to the total number of transmissions, is the historical average transmission rate index, is the ratio of the historical average transmission rate to the device’s rated transmission rate, and W 1 , W 2is the influence weight of the historical transmission error rate and the historical average transmission rate on the performance evaluation index value. After obtaining the historical transmission error rate and the historical average transmission rate of the target device that has not completed data synchronization based on the historical big data, the historical transmission error rate and the historical average transmission rate are input into the performance evaluation index estimation model to obtain the performance evaluation index, and then the target device that has not completed data synchronization is divided into the first target device group and the second target device group according to the performance evaluation index of the target device that has not completed data synchronization.

[0035] It should be noted that, in order to facilitate the estimation of the performance evaluation index, the inventor regards the historical transmission error rate and the historical average transmission rate of the target device as being of equal importance, that is, The values ​​are all assigned to 0.5. In the performance evaluation index prediction model expression, the applicant introduced an exponential function for the transmission rate part during the data fitting process. , so that when the transmission rate increases, its contribution to PEI will increase exponentially; this reflects that in practical applications, high-speed transmission often has higher value than low-speed transmission.

[0036] In one embodiment, dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group according to the performance evaluation index of the target devices for which data synchronization has not been completed includes: Sorting the target devices that have not completed data synchronization from large to small according to performance evaluation indexes to obtain a target device queue; A preset number of target devices with uncompleted data synchronization are acquired in reverse order from the tail of the target device queue to obtain a second target device group, and the rest are divided into a first target device group.

[0037] Specifically, the target device queue is obtained by sorting the target device data transmission according to the overall performance, a part of the number is obtained from the head of the target device queue to obtain the first target device group, and a part of the number is obtained from the tail of the target device queue to obtain the second target device group. For example, if the number of target devices in the target device queue is 10, the top 5 target devices with the overall performance of data transmission can be selected as the first target device group, and the bottom 5 target devices with the overall performance of data transmission can be selected as the second target device group. Since the second shared source device is a shared source device obtained after data repair, its performance is relatively low compared to the first shared source device. Therefore, the present application uses the first shared source device to perform data sharing synchronization on high-performance target devices, and uses the second shared source device to perform data sharing synchronization on low-performance target devices. By matching the high-performance first shared source device with the high-performance target device, the high-speed processing capabilities of both parties can be fully utilized to achieve high-speed data transmission and synchronization. At the same time, equipping the low-performance target device with a suitable shared source device can ensure the stability of data transmission and avoid inefficiency or errors caused by performance mismatch; and the present application helps to reasonably allocate resources to avoid waste of resources when high-performance devices process data of low-performance target devices, while also ensuring that the low-performance target device can obtain sufficient data support.

[0038] Furthermore, since the second shared source device is a shared source device obtained after data repair, its performance is relatively low. Therefore, the effect of using the second shared source device for data sharing synchronization needs to be further confirmed, that is, the shared data of the second target device group can be verified. In the case where the shared data verification of the second target device group fails, the shared data request is re-issued. In other words, after the second shared source device is used to perform data sharing synchronization on the second target device group, the shared data of the second target device group needs to be verified to ensure that the data of the second target device group is successfully shared and synchronized.

[0039] S600: The first sharing source device continues to complete data sharing synchronization of the first target device group, and the second sharing source device completes data sharing synchronization of the second target device group after receiving the data synchronization authority.

[0040] It is understandable that when the target devices that have not completed data synchronization are randomly divided in step 500, due to the different states of each target device, there is a large difference in the data sharing efficiency of the first target device group and the second target device group. Therefore, in the data sharing synchronization process, data sharing balance can be performed according to the synchronization state of the first sharing source device and the second sharing source device. In one embodiment, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority, including: S610: Establishing a communication link between the first sharing source device and the second sharing source device to exchange data synchronization status information about the target device groups that each device is responsible for in real time; S620: when the data synchronization progress of the first target device group is greater than a preset multiple of the data synchronization progress of the second target device group, transfer the target devices in the second target device group from the current group to the first target device group to achieve a balance in data sharing; S630: When the data synchronization progress of the second target device group is greater than a preset multiple of the data synchronization progress of the first target device group, transfer the target devices in the first target device group from the current group to the second target device group to achieve a balance in data sharing.

[0041] Specifically, by establishing a communication link between the first shared source device and the second shared source device to exchange data synchronization status information about the target device group for which each is responsible in real time, the overall data synchronization efficiency and accuracy are optimized. When any shared source device detects a data synchronization delay or error in the target device group for which it is responsible, it can notify the other party through the communication link and negotiate to adjust the data synchronization strategy to restore the data consistency of all target devices as quickly as possible. For example, when the data synchronization progress of a certain target device group is significantly faster or slower than that of other device groups, the system can automatically or according to user instructions transfer the device from the current group to another group, so as to more effectively utilize the processing power of each shared source device and achieve load balancing of data synchronization. The synchronization progress can be determined by the number of devices that complete synchronization at the same time.

[0042] In other embodiments, the first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority, including: Predict the data synchronization completion time of each target device based on a pre-trained machine learning prediction model; When the proportion of target devices whose synchronization completion time is greater than the preset time is greater than the preset proportion value, the target devices that have completed data synchronization in the first target device group or the second target device group are added as third sharing source devices.

[0043] Specifically, firstly, the data synchronization completion time of each target device is predicted based on the pre-trained machine learning prediction model; when the system detects that the proportion of target devices whose synchronization completion time is greater than the preset time is greater than the preset proportion value, it means that the data synchronization of most target devices is slow. At this time, the system automatically triggers the resource optimization mechanism, and selects a target device with superior performance and sufficient resources from the first target device group or the second target device group that has completed data synchronization, and adds it as the third shared source device. Specifically, the newly added third shared source device will assist the original first shared source device and the second shared source device to jointly undertake the data synchronization task. The system will intelligently assign data synchronization tasks to the third shared source device based on the current load conditions, performance parameters and remaining available resources of each shared source device to balance the overall load, reduce data synchronization delays, and further improve the efficiency and reliability of the entire data synchronization system.

[0044] In other embodiments, the data synchronization priority of each target device can also be adjusted according to the data synchronization completion time of each target device, wherein the target device with a short data synchronization completion time is adjusted to a high priority. In other words, the target device that is easier to complete data synchronization is given priority to complete data sharing synchronization. In this way, by giving priority to the target device that is easier to complete data synchronization, the number of unsynchronized devices in the system can be quickly reduced, thereby speeding up the progress of the entire data sharing synchronization process.

[0045] Based on this, the data sharing method based on massive data of the present application first obtains the target shared device group based on the data synchronization instruction, monitors the data synchronization completion status of the target shared device group in real time and obtains the second shared source device during the process of sending shared data to the target shared device group, and then judges the consistency between the shared data of the second shared source device and the shared data of the first shared source device; when the shared data of the second shared source device is inconsistent with the shared data of the first shared source device, the shared data of the second shared source device is repaired so that the second shared source device has the data sharing function; then the target devices that have not completed data synchronization are divided into the first target device group and the second target device group, and the data synchronization authority of the second target device group is allocated to the second shared source device; finally, the first shared source device continues to complete the data sharing synchronization of the first target device group, and the second shared source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority. The system aims to solve the many challenges faced by existing data sharing systems when faced with massive data by introducing innovative technologies such as multi-source sharing, intelligent scheduling and repair mechanisms, and dynamic performance evaluation indexes. By optimizing data synchronization strategies, improving data consistency and reliability, and achieving flexible allocation of resources, the system provides strong support for efficient and reliable data sharing in massive data environments.

[0046] The present application also provides a data sharing information system based on massive data. Figure 3 , Figure 3 A schematic diagram of the hardware structure of a data sharing information system based on massive data provided for some embodiments of the present application; comprising a first shared source device, a memory 110, and a processor 120, wherein the memory 110 is used to store program code, and the processor 120 is used to call the program code to execute the method described above.

[0047] Among them, the processor 120 is used to provide computing and control capabilities to control the data sharing information system based on massive data to perform corresponding tasks, for example, to control the data sharing information system based on massive data to perform the data sharing method based on massive data in any of the above method embodiments, the method comprising: a first sharing source device obtains a target sharing device group based on a data synchronization instruction, the target sharing device group includes a plurality of target devices to be synchronized with data; in the process of sending shared data to the target sharing device group, real-time monitoring of the data synchronization completion status of the target sharing device group and obtaining a second sharing source device, the second sharing source device is a target device that receives the shared data and completes data synchronization; judging the second the consistency between the shared data of the shared source device and the shared data of the first shared source device; in the case that the shared data of the second shared source device is inconsistent with the shared data of the first shared source device, repairing the shared data of the second shared source device so that the second shared source device has a data sharing function; dividing the target devices that have not completed data synchronization into a first target device group and a second target device group, and allocating the data synchronization authority of the second target device group to the second shared source device; the first shared source device continues to complete the data sharing synchronization of the first target device group, and the second shared source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority.

[0048] The processor 120 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0049] The memory 110, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the data sharing method based on massive data in the embodiments of the present application. The processor 120 can implement the data sharing method based on massive data in any of the above method embodiments by running the non-transitory software programs, instructions and modules stored in the memory 110.

[0050] Specifically, the memory 110 may include a volatile memory (VM), such as a random access memory (RAM); the memory 110 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD) or other non-transitory solid-state storage device; the memory 110 may also include a combination of the above types of memory.

[0051] To sum up, the data sharing information system based on massive data of the present application adopts the technical solution of any one of the above-mentioned data sharing method embodiments based on massive data. Therefore, it has at least the beneficial effects brought by the technical solutions of the above-mentioned embodiments, which will not be repeated here one by one.

[0052] The embodiment of the present application also provides a computer-readable storage medium, such as a memory including a program code, and the program code can be executed by a processor to complete the data sharing method based on massive data in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0053] The present application also provides a computer program product, which includes one or more program codes, which are stored in a computer-readable storage medium. A processor of a data sharing information system based on massive data reads the program code from the computer-readable storage medium, and the processor executes the program code to complete the steps of the data sharing method based on massive data provided in the above embodiment.

[0054] A person skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be accomplished by hardware or by hardware associated with a program code, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0055] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0056] Through the description of the above embodiments, it can be clearly understood by those skilled in the art that each embodiment can be implemented by means of software plus a general hardware platform, or by hardware. It can be understood by those skilled in the art that all or part of the processes in the above embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0057] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A data sharing information system based on massive data, characterized in that: The system comprises a first shared source device, a memory and a processor, wherein the memory stores executable program code, and the processor is used to implement the following steps when calling and running the executable program code: The first sharing source device acquires a target sharing device group based on a data synchronization instruction, wherein the target sharing device group includes a plurality of target devices to be synchronized with data; In the process of sending shared data to the target shared device group, real-time monitoring of the data synchronization completion status of the target shared device group is performed and a second shared source device is obtained, where the second shared source device is a target device that receives the shared data and completes data synchronization; determining consistency between the shared data of the second sharing source device and the shared data of the first sharing source device; When the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, repairing the shared data of the second sharing source device so that the second sharing source device has a data sharing function; Dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group, and allocating the data synchronization authority of the second target device group to the second sharing source device; The first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority.

2. The data sharing information system based on massive data as claimed in claim 1, characterized in that: The step of dividing the target devices that have not completed data synchronization into a first target device group and a second target device group includes: Based on historical big data, the historical transmission error rate and historical average transmission rate of the target device that has not completed data synchronization are obtained; Obtaining a performance evaluation index of the target device that has not completed data synchronization according to the historical transmission error rate and the historical average transmission rate, wherein the performance evaluation index is used to characterize the overall performance of data transmission of the target device; The target devices for which data synchronization has not been completed are divided into a first target device group and a second target device group according to the performance evaluation indexes of the target devices for which data synchronization has not been completed, wherein the sum of the performance evaluation indexes of the first target device group is greater than the sum of the performance evaluation indexes of the second target device group.

3. The data sharing information system based on massive data as claimed in claim 2, characterized in that: The obtaining of the performance evaluation index of the target device for which data synchronization has not been completed according to the historical transmission error rate and the historical average transmission rate includes: The historical transmission error rate and the historical average transmission rate are input into a pre-trained performance evaluation index prediction model to obtain a performance evaluation index value, and the performance evaluation index prediction model satisfies the following expression: ； Among them, ER is the historical transmission error rate, which is the ratio of the number of historical transmission errors to the total number of transmissions. is the historical average transmission rate index, is the ratio of the historical average transmission rate to the device calibrated transmission rate, W1 and W2 are the influence weights of the historical transmission error rate and the historical average transmission rate on the performance evaluation index value.

4. The data sharing information system based on massive data as claimed in claim 3, characterized in that: The step of dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group according to the performance evaluation index of the target devices for which data synchronization has not been completed includes: Sorting the target devices that have not completed data synchronization from large to small according to performance evaluation indexes to obtain a target device queue; A preset number of target devices with uncompleted data synchronization are acquired in reverse order from the tail of the target device queue to obtain a second target device group, and the rest are divided into a first target device group.

5. The data sharing information system based on massive data as claimed in claim 4, characterized in that: After the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization permission, the method further includes: The shared data of the second target device group is verified, and if the shared data of the second target device group fails to be verified, a shared data request is made again.

6. The data sharing information system based on massive data as claimed in claim 1, characterized in that: The first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority, including: Establishing a communication link between the first sharing source device and the second sharing source device to exchange data synchronization status information about the target device groups that each device is responsible for in real time; When the data synchronization progress of the first target device group is greater than a preset multiple of the data synchronization progress of the second target device group, the target devices in the second target device group are transferred from the current group to the first target device group to achieve a balance in data sharing; When the data synchronization progress of the second target device group is greater than a preset multiple of the data synchronization progress of the first target device group, the target devices in the first target device group are transferred from the current group to the second target device group to achieve a balance in data sharing.

7. The data sharing information system based on massive data as claimed in claim 1, characterized in that: The first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority, including: Predict the data synchronization completion time of each target device based on a pre-trained machine learning prediction model; When the proportion of target devices whose synchronization completion time is greater than the preset time is greater than the preset proportion value, the target devices that have completed data synchronization in the first target device group or the second target device group are added as third sharing source devices.

8. The data sharing information system based on massive data as claimed in claim 7, characterized in that: After predicting the data synchronization completion time of each target device based on the pre-trained machine learning prediction model, the method further includes: The data synchronization priority of each target device is adjusted according to the data synchronization completion time of each target device, wherein the data synchronization completion time is adjusted to a high priority.

9. The data sharing information system based on massive data as claimed in claim 1, characterized in that: The repairing of the shared data of the second sharing source device so that the second sharing source device has a data sharing function includes: Performing sequence adjustment and repair on the shared data of the second sharing source device based on a shared data sequence adjustment trigger; A secondary check is performed on the shared data after the sequence adjustment and repair, and a rollback recovery of the shared data is performed when the sequence adjustment and repair fails.

10. A data sharing method based on massive data, characterized in that: The method comprises: The first sharing source device acquires a target sharing device group based on a data synchronization instruction, wherein the target sharing device group includes a plurality of target devices to be synchronized with data; In the process of sending shared data to the target shared device group, real-time monitoring of the data synchronization completion status of the target shared device group is performed and a second shared source device is obtained, where the second shared source device is a target device that receives the shared data and completes data synchronization; determining consistency between the shared data of the second sharing source device and the shared data of the first sharing source device; When the shared data of the second sharing source device is inconsistent with the shared data of the first sharing source device, repairing the shared data of the second sharing source device so that the second sharing source device has a data sharing function; Dividing the target devices for which data synchronization has not been completed into a first target device group and a second target device group, and allocating the data synchronization authority of the second target device group to the second sharing source device; The first sharing source device continues to complete the data sharing synchronization of the first target device group, and the second sharing source device completes the data sharing synchronization of the second target device group after receiving the data synchronization authority.

Images