Cloud computing platform data storage method based on big data analysis

Abstract

The invention relates to the technical field of cloud computing data storage, and discloses a cloud computing platform data storage method based on big data analysis, and the method comprises the steps: estimating the data transmission rate of each data synchronization link; executing a duration cost verification strategy, estimating the transmission duration of the data packet in the data synchronization link according to the length of the data packet, and calculating the synchronization verification duration of the data packet to set two types of data transmission strategies; executing a single-path direct-write verification strategy, and controlling the execution states of the two types of data transmission strategies according to a result of verifying the integrity of the data packet by the main storage database in one type of strategy; executing a database reconstruction strategy, jointly processing data packets of the database and verifying the data packets of the database, upgrading non-key data based on the distribution density of key data, and reconstructing records of user behavior data; and transferring the current data packet to a buffer database determined according to the data synchronization link for temporary storage. And the data storage system has self-adaption, fault tolerance and high efficiency.

Description

Technical Field

[0001] This invention relates to the field of cloud computing data storage technology, specifically to a cloud computing platform data storage method based on big data analysis. Background Technology

[0002] With the development of internet applications and e-commerce, cross-platform online marketing heavily relies on the continuous collection and analysis of user behavior data. By centrally collecting and analyzing user clicks, browsing, and purchasing behaviors across different platforms, marketing effectiveness can be evaluated, advertising strategies optimized, resource utilization efficiency improved, and user conversion rates increased, thereby enhancing market competitiveness.

[0003] Existing user behavior data storage solutions typically use data packets as the smallest processing unit, employing fixed data transmission and verification processes without differentiating the importance of user behavior data. When link status changes or data volume fluctuates, all data is transmitted, verified, and written in a fixed order, lacking flexible scheduling. This results in data processing timeliness being constrained by the slowest link or the longest verification path, making overall write latency difficult to control. Furthermore, existing solutions lack data quality adjustment mechanisms based on the distribution characteristics of user behavior data. When packet loss, verification anomalies, or link instability occur, targeted protection or structural adjustments for critical data are impossible, easily leading to sparse distribution, fragmented records, or delayed writes of critical user behavior data in the stored results. If data packets are lost, parsing fails, or origin retrieval fails during transmission or processing, existing technologies typically rely on overall retransmission or batch origin retrieval, lacking flexible recovery strategies for critical data. This increases system load and makes it difficult to promptly restore a continuous and reliable data storage state.

[0004] This solution proposes a data storage method for cloud computing platforms based on big data analytics. Summary of the Invention

[0005] This invention provides a data storage method for a cloud computing platform based on big data analytics, which helps to solve the problems mentioned in the background art.

[0006] This invention provides the following technical solution: a data storage method for a cloud computing platform based on big data analytics, comprising:

[0007] User behavior data is recorded in the database, and the user behavior data is evenly divided into data fragments and encrypted and encapsulated into data packets.

[0008] The verification database and processing database each establish independent data synchronization links with the acquisition database to receive the data packets and send them to the main storage database after verification. Specifically:

[0009] Estimate the data transmission rate of each data synchronization link;

[0010] The processing database completes the full reception of the data packet before the verification database;

[0011] The execution time cost verification strategy estimates the transmission time of data packets on the data synchronization link based on the data packet length, and calculates the synchronization verification time for the data packets in order to set two types of data transmission strategies;

[0012] The two types of data transmission strategies are: one strategy in which the processing database sends data packets to the main storage database alone, and the other strategy in which the processing database and the verification database jointly send data packets to the main storage database.

[0013] The single-path write-through verification strategy is executed, and the execution status of the two data transmission strategies is controlled based on the result of the main storage database's data packet integrity verification in one type of strategy.

[0014] When executing a type II policy:

[0015] Implement database reconstruction strategies, jointly process and verify database data packets, upgrade non-critical data based on the distribution density of critical data, and reconstruct user behavior data records;

[0016] The current data packet is temporarily stored in a buffer database determined based on the data synchronization link.

[0017] Optionally, estimating the data transmission rate of each data synchronization link includes:

[0018] For any data synchronization link:

[0019] Set time window , obtain in Number of data packets attempted to be sent by the data synchronization link in the middle ;

[0020] Get in Number of data packets successfully received by the data synchronization link ;

[0021] Calculate the packet loss rate of the data synchronization link. ;

[0022] Calculate retransmission coefficient , These represent the number of retransmissions and the number of packet losses within the time window, respectively. Each lost packet can correspond to one or more retransmissions.

[0023] Estimate the data transmission rate of the data synchronization link. The data transmission rate is used to estimate the transmission time of data packets in the data synchronization link.

[0024] Optionally, the execution time cost verification strategy estimates the transmission time of the data packet on the data synchronization link based on the data packet length and calculates the synchronization verification time for the data packet, including:

[0025] Obtain the length of the untransmitted link in the data synchronization link between the verification database and the acquisition database. ;

[0026] Get the length of a single data packet ,calculate , The total length of the data packet;

[0027] Calculate the time elapsed between the data packet acquisition database and the verification database. ;

[0028] The data packet verification includes integrity verification and synchronization verification. The main storage database performs integrity verification on the data packets, while the processing database and the verification database perform synchronization verification on the data packets.

[0029] Calculate the historical synchronization verification duration of the verification database. To retrieve any single data packet from the past;

[0030] The data packet is parsed into records of individual user behavior data, the number of which is... ;

[0031] For any record: Calculate the record complexity , For the standard length of Bitcoin records, These are the weighting coefficients. Bitcoin length The parsing complexity of the data packets;

[0032] Get the verification capabilities of the verification database Calculate the duration of synchronous verification. ;

[0033] The user behavior data is divided into critical data and non-critical data. The average time for synchronous verification of critical data and non-critical data is calculated and denoted as follows: and ;

[0034] Get the number of critical and non-critical data in the data packet being parsed. and ;

[0035] calculate .

[0036] Optionally, the method for setting two types of data transmission strategies includes:

[0037] The link length from the processing database to the main storage database is obtained. ;

[0038] Calculate the transmission time of data packets sent from the processing database to the main storage database. ;

[0039] like Configure the processing database to send data packets separately to the main storage database;

[0040] like The configuration involves the processing database and the verification database synchronizing and verifying the data packets before sending them to the main storage database.

[0041] Optionally, the execution of the single-path write-through verification strategy controls the execution status of the two data transmission strategies based on the result of the main storage database's data packet integrity verification in one strategy, including:

[0042] For any data packet obtained by the main storage database from the processing database, user behavior data, which is considered key data, is extracted from the data packet and substituted into the CRC function as a parameter to obtain the checksum field. ;

[0043] Obtain the standard verification fields encapsulated in the data packet itself. Set the verification error threshold ;

[0044] like If the integrity verification passes, the main storage database sends a shutdown synchronization verification process to the processing database.

[0045] The verification database stops receiving data packets from the acquisition database and does not perform synchronization verification;

[0046] like If the integrity verification fails, the data packet that fails will be recorded as an abnormal data packet.

[0047] Obtain the link length from the data collection database to the processing database. Calculate the time to retrieve data from the source. ;

[0048] like Then the main storage database will return the abnormal data packets in batches to the processing database, and the processing database will request replacement abnormal data packets from the collection database.

[0049] If the attempt to replace the abnormal data packet at the source fails, then the second-class strategy will be executed.

[0050] If the replacement of the abnormal data packet is successful, the replacement abnormal data packet will be sent to the main storage database.

[0051] Optionally, the implementation of the database reconstruction strategy, jointly processing data packets of the database and verifying the database, and performing critical data upgrades on non-critical data based on the critical data distribution density, includes:

[0052] When the data packets received from the processing database and the verification database correspond to the same user behavior data and are critical data, synchronous verification is performed on the single piece of user behavior data:

[0053] Retrieve records of single user behavior data from two sources, determine if the primary keys are the same, and compare the hash values ​​of the same fields in the records if the primary keys are the same;

[0054] If the hash values ​​are the same, the synchronization verification passes. All user behavior data from this verification are merged to obtain a synchronization verification table, and the distribution density of key data is calculated. , For the amount of key data, The total number of user behavior data;

[0055] Set key data density thresholds ;

[0056] like Retrieve the index of any adjacent key data in the synchronization check table. ;

[0057] Calculate the index interval ;

[0058] Set the upper limit of the index interval ,like Then in key data and Filter out non-critical data and upgrade the non-critical data to critical data;

[0059] The index for non-critical data used for upgrades is ,

[0060] Send the upgraded synchronization checklist to the main storage database.

[0061] Optionally, the implementation of the database reconstruction strategy, the joint processing of data packets for the database and the verification database, and the performance of critical data upgrades on non-critical data based on the critical data distribution density, further includes:

[0062] If the synchronization check fails, determine whether the primary key of the critical data can be mapped.

[0063] If the primary key of the key data can be mapped, the key data is collected from the collection database according to the mapped primary key.

[0064] If the primary key of critical data cannot be mapped, the control acquisition database randomly retransmits data packets to the verification database, and the verification database determines whether the retransmitted data packets are missing.

[0065] Get the number of retransmitted data packets In the retransmitted data packets, the number of times the database's missing critical data was checked was [number missing]. The number of missing non-keywords is ;

[0066] Calculate missing value , and These are the weighting coefficients;

[0067] Set missing threshold ;

[0068] like The merged synchronization check table is sent to the main storage database.

[0069] like If so, the data packet will be sent to the buffer database.

[0070] Optionally, transferring the current data packet to a buffer database determined according to the data synchronization link for temporary storage includes:

[0071] Obtain any database that does not participate in writing to the main storage database, and calculate the transmission time for establishing a data synchronization link between the database and the verification database and the processing database;

[0072] The sum of the transmission durations is denoted as the transmission cost;

[0073] The database with the lowest transmission cost is used as the buffer database, and the verification database exists. When this happens, the data packets currently stored in the verification database are transferred to the buffer database.

[0074] The present invention has the following beneficial effects:

[0075] 1. This cloud computing platform data storage method based on big data analytics uniformly divides user behavior data into data fragments and encrypts and encapsulates them into data packets. It establishes independent data synchronization links between the acquisition database, processing database, and verification database, enabling parallel reception and verification. This avoids data loss due to single-link failures and improves the reliability and timeliness of data transmission. The introduction of a processing database receiving data first ensures that critical data can be stored intact in the shortest possible time, guaranteeing real-time analysis and response to core business indicators (such as purchasing behavior). Simultaneously, by setting a time-cost verification strategy and two types of data transmission strategies, it flexibly balances the principles of time priority and quality priority. While ensuring the priority storage of critical data, the two strategies are used to reconstruct and upgrade non-critical data, improving overall data integrity and verification reliability. The introduction of a buffer database ensures that data is not lost in abnormal situations, providing a guarantee for subsequent source retrieval or retransmission, making the data storage system adaptive, fault-tolerant, and efficient.

[0076] 2. This cloud computing platform data storage method based on big data analytics calculates the data transmission time from the collection database to the verification database, the total length of data packets, and the historical synchronization verification time, enabling quantitative management of cross-platform data synchronization performance. Data packets are parsed into individual user behavior records, and the complexity of each record is calculated. Dynamic estimation of synchronization verification time is performed for different data types and complexities, ensuring optimal timeliness of critical data during transmission and verification. Synchronization verification distinguishes between critical and non-critical data, effectively reducing the processing burden of non-critical data while ensuring the accuracy and integrity of core business data. It adapts to online marketing scenarios with large data volumes and uneven distribution of user behavior, enabling timely processing of critical behaviors (such as purchase actions) while deferred processing of non-critical behaviors such as browsing and clicking. This ensures the accuracy of marketing decisions while reducing system load and improving data processing efficiency.

[0077] 3. This cloud computing platform data storage method based on big data analytics calculates the transmission time from the processing database to the main storage database. Combined with the data transmission and verification time from the acquisition database to the verification database, it dynamically selects between a single-path direct write strategy and a joint synchronization strategy. This ensures that critical data is written in the shortest possible time, while maintaining data integrity and consistency when time permits. The single-path direct write strategy allows key data, such as core purchasing behavior data received first by the processing database, to be quickly updated to the main storage database, ensuring that marketing decisions such as advertising placement and inventory adjustments take effect in real time. The joint synchronization strategy, when time allows, improves data reliability through synchronous verification between the processing and verification databases. The main storage database performs integrity verification on data packets. Origin tracking mechanisms and abnormal data handling promptly identify abnormal data packets and take measures such as origin requests, secondary strategies, or buffer database temporary storage to complete critical data replenishment, ensuring the integrity and availability of core business data.

[0078] 4. This cloud computing platform data storage method based on big data analytics dynamically assesses the integrity and density of critical data within data packets by performing synchronous verification on key data and calculating its distribution density. When density is insufficient or index intervals are too large, adjacent non-critical data is upgraded to critical data. Since complete verification and synchronous verification only target critical data, this enhances the reliability of data verification. It balances timeliness and data integrity, enabling rapid synchronization of core business data while adding redundant verification through the upgrading of non-critical data, thus improving cross-platform data consistency. It significantly reduces the risk of critical data loss due to network packet loss or origin server failures, improves the accuracy and reliability of cross-channel user behavior analysis, and ensures that non-critical data is optimized for use when necessary.

[0079] 5. This cloud computing platform data storage method based on big data analytics utilizes a buffer database to prevent delays or errors in main storage database updates caused by link loss, retransmission failures, or missing data packets. It possesses secondary compensation capabilities, improving overall data transmission reliability and system fault tolerance. Simultaneously, through quantitative optimization of transmission costs, it ensures that the optimal path for storing and forwarding data packets can be selected under resource-constrained or high-load environments, achieving efficient management and stable storage of cross-platform user behavior data. Attached Figure Description

[0080] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] Example 1, refer to Figure 1 A data storage method for a cloud computing platform based on big data analytics, comprising:

[0083] E-commerce companies operate official websites, mobile apps, and WeChat mini-programs simultaneously. Recently, they have been conducting promotional activities and are tracking user purchasing behavior in real time to determine advertising placement and inventory allocation. Cross-platform user behavior data shows that users first browse products on the website and add them to their shopping carts, then click to claim coupons on the app, browse other products, and finally complete the purchase and place the order on the WeChat mini-program.

[0084] The data volume is large and unevenly distributed: website browsing and click behavior generates a large amount of data (non-critical data), about 1,000 records per minute, while mini-program purchase behavior (critical data) is less numerous but crucial for advertising decisions and inventory management, with about 50 records per minute.

[0085] Link fluctuations and packet loss issues: During peak network hours, the packet loss rate for the app reaches 7%, requiring some data packets to be retransmitted. While website and mini-program transmissions are stable, critical purchase data is delayed by 50-100ms during synchronization to the main storage database due to link latency. If all data is synchronized using a unified strategy, packet loss or delays can lead to the delayed writing or loss of critical purchase data, impacting real-time ad delivery and promotional strategy adjustments. For example, a merchant's promotion may have ended, but different platforms may still display it as valid. Users viewing the promotion on one platform may mistakenly believe it is still ongoing, leading to transaction discrepancies or customer dissatisfaction.

[0086] This solution prioritizes purchasing activities (critical data) to ensure that critical data is written to the main storage quickly and securely; non-critical data is downgraded or written with a delay to reduce system load.

[0087] Results: In cross-platform online marketing scenarios, the overall write latency of key purchase behavior data from the collection database to the main storage database is approximately 300 ms (302.2 ms). Compared to the write latency of over 500 ms under a unified synchronization strategy, the timeliness is improved by about 40%. Non-critical behavior data such as browsing and clicking can be delayed in writing or temporarily stored in the buffer database when the link fluctuates or the verification is abnormal. This reduces the non-critical verification load by about 70% during the synchronization of critical data and avoids the blocking of critical writes.

[0088] During high-concurrency marketing campaigns (approximately 1000 views / clicks / minute and 50 purchases / minute), the instantaneous write pressure on the main storage database decreased by about 30%, and the continuity of key purchase data in the main storage remained above 95%. Cross-platform user purchase behavior can be aggregated and analyzed within 1 second, supporting real-time adjustments to advertising strategies and inventory allocation.

[0089] E-commerce companies collect user behavior data from their websites, apps, and WeChat mini-programs, including browsing, clicking, and purchasing behaviors, and store this data in a database to generate user behavior data. Each record includes user ID, platform, behavior type, product ID, and timestamp.

[0090] Key data: purchasing behavior; non-key data: browsing and clicking behavior. The packaged data is 400KB in size and contains both key and non-key data. The data package is sent to the verification database and the processing database in parallel.

[0091] In this approach, the distinction between key data is not fixed but dynamically determined based on specific online marketing strategies and business objectives. For example, for e-commerce platforms, user behavior data typically includes browsing, clicking, and purchasing behavior. Purchasing behavior is directly related to transaction success and ad conversion rates, and is therefore defined as key data. Browsing or clicking behavior, on the other hand, can be considered non-key data, used only for subsequent analysis or behavioral modeling. The definition of key data can be adjusted under different marketing strategies.

[0092] Sending two identical copies of data simultaneously is not for redundant storage, but to establish a verifiable and traceable parallel perspective on the same business fact without blocking business processing; otherwise, the system will be unable to provide a credible conclusion on the authenticity and completeness of the processing results.

[0093] The data acquisition database, processing database, verification database, and main storage database are all deployed on the same cloud computing platform, leveraging the high-performance computing and storage capabilities of the cloud platform to achieve centralized data management across platforms and channels.

[0094] While the primary storage database can accept direct writes, it is mainly responsible for core storage and business queries. If direct writes are performed, network fluctuations or transmission anomalies may lead to the loss of critical data or incomplete writes. Under high concurrency, the write pressure is high, which may lead to transaction failures or delays. It is impossible to independently verify the integrity of data before writing, which increases business risks.

[0095] Estimate the data transmission rate of each data synchronization link;

[0096] For website data synchronization: Set a time window , obtain in Number of data packets attempted to be sent by the data synchronization link in the middle ;

[0097] Get in Number of data packets successfully received by the data synchronization link ;

[0098] Calculate the packet loss rate of the data synchronization link. ;

[0099] Calculate retransmission coefficient , These represent the number of retransmissions and the number of packet losses within the time window, respectively. Each lost packet can be retransmitted once or multiple times.

[0100] Estimate the data transmission rate of the data synchronization link. packets per second The data transmission rate is used to estimate the transmission time of data packets in the data synchronization link.

[0101] The processing database completes the full reception of the data packet before the verification database;

[0102] The execution time cost verification strategy estimates the transmission time of data packets on the data synchronization link based on the data packet length, and calculates the synchronization verification time for the data packets in order to set two types of data transmission strategies;

[0103] Full verification and synchronous verification are mainly performed on the most important core data to ensure the integrity and timeliness of data during transmission and storage.

[0104] This embodiment focuses on the website platform:

[0105] Obtain the length of the untransmitted link in the data synchronization link between the verification database and the acquisition database. ;

[0106] Get the length of a single data packet ,calculate , The total length of the data packet is [value]. The total lengths of the data packets for the App and WeChat Mini Program are respectively [values]. ;

[0107] Calculate the time elapsed between the data packet acquisition database and the verification database. ; Each packet is 400KB per second, and the transmission rate is [missing information]. ;

[0108] The data packet verification includes integrity verification and synchronization verification. The main storage database performs integrity verification on the data packets, while the processing database and the verification database perform synchronization verification on the data packets.

[0109] Calculate the historical synchronization verification duration of the verification database. To retrieve any single data packet from the past;

[0110] The data packet is parsed into records of individual user behavior data, the number of which is... ;

[0111] For any record: Calculate the record complexity , For the standard length of Bitcoin records, These are the weighting coefficients. Bitcoin length The parsing complexity of the data packets;

[0112] Get the verification capabilities of the verification database Calculate the duration of synchronous verification. ;

[0113] In the website platform, the user behavior data is divided into key data and non-key data. The average time for synchronous verification of key data and non-key data is calculated and denoted as follows: and ;

[0114] The number of critical and non-critical data in the data packet being parsed this time. and ;

[0115] calculate ;

[0116] The three platforms are websites: ;

[0117] The link length from the website platform's processing database to the main storage database is obtained. ;

[0118] Calculate the transmission time of data packets sent from the processing database to the main storage database. ;

[0119] The processing database sends data packets to the main storage database separately;

[0120] When the processing database receives the target data packet before the verification database, and based on a comprehensive evaluation of the link transmission time and historical verification time, determines that synchronous verification would significantly increase the overall processing latency, the processing database will no longer wait for the verification database to complete synchronous verification. Instead, it will directly send the data packet to the main storage database. The main storage database performs integrity verification on the data packet and prioritizes storing user behavior data identified as critical data within the data packet.

[0121] The single-path write-through verification strategy is executed, and the execution status of the two data transmission strategies is controlled based on the result of the main storage database's data packet integrity verification in one type of strategy.

[0122] For any data packet obtained by the main storage database from the processing database, user behavior data, which is considered key data, is extracted from the data packet and substituted into the CRC function as a parameter to obtain the checksum field. ;

[0123] Obtain the standard verification fields encapsulated in the data packet itself. Set the verification error threshold ;

[0124] like If the integrity verification passes, the main storage database sends a shutdown synchronization verification process to the processing database.

[0125] The verification database stops receiving data packets from the acquisition database and does not perform synchronization verification;

[0126] like If the integrity verification fails, the data packet that fails will be recorded as an abnormal data packet.

[0127] Obtain the link length from the data collection database to the processing database. Calculate the time to retrieve data from the source. ;

[0128] like Then the main storage database will return the abnormal data packets in batches to the processing database, and the processing database will request replacement abnormal data packets from the collection database.

[0129] If the attempt to replace the abnormal data packet at the source fails, then the second-class strategy will be executed.

[0130] If the replacement of the abnormal data packet is successful, the replacement abnormal data packet will be sent to the main storage database.

[0131] In this embodiment, Implement the second type of strategy;

[0132] When the main storage database determines that a data packet is abnormal during the integrity verification process, this invention also prioritizes attempting to obtain a replacement data packet from the acquisition database through a back-to-source mechanism, based on the principle of time cost.

[0133] The triggering, execution, and batch return processes of the source data replenishment are all aimed at minimizing the latency of acquiring critical data, thereby ensuring that critical user behavior data can be completed and stored in the shortest possible time.

[0134] like The configuration involves the processing database and the verification database synchronizing and verifying the data packets before sending them to the main storage database.

[0135] When the evaluation results show that the single-path direct write strategy cannot reliably write critical data within the expected time, or that integrity verification and source repackaging fail to meet the critical data consistency requirements, this invention automatically switches to a quality-first data synchronization verification strategy.

[0136] Under the quality-first strategy, the processing database and the verification database perform synchronous verification on data packets. During the synchronous verification process, non-critical user behavior data is reconstructed and upgraded to enhance the coverage and reliability of data verification.

[0137] The two types of data transmission strategies are: one strategy in which the processing database sends data packets to the main storage database alone, and the other strategy in which the processing database and the verification database jointly send data packets to the main storage database.

[0138] When executing a type II policy:

[0139] Implement database reconstruction strategies, jointly process and verify database data packets, upgrade non-critical data based on the distribution density of critical data, and reconstruct user behavior data records;

[0140] When the data packets received from the processing database and the verification database correspond to the same user behavior data and are critical data, synchronous verification is performed on the single piece of user behavior data:

[0141] Retrieve records of single user behavior data from two sources, determine if the primary keys are the same, and compare the hash values ​​of the same fields in the records if the primary keys are the same;

[0142] If the hash values ​​are the same, the synchronization verification passes. All user behavior data from this verification are merged to obtain a synchronization verification table, and the distribution density of key data is calculated. , For the amount of key data, The total number of user behavior data;

[0143] Set key data density thresholds ;

[0144] like Retrieve the index of any adjacent key data in the synchronization check table. ;

[0145] Calculate the index interval ;

[0146] Set the upper limit of the index interval , Then in key data and Filter out non-critical data and upgrade the non-critical data to critical data;

[0147] The index for non-critical data used for upgrades is , ;

[0148] Send the upgraded synchronization checklist to the main storage database.

[0149] Specifically, when the distribution density of critical data in data packets is low, relying solely on critical data for synchronous verification can easily lead to insufficient verification coverage, thereby affecting the reliability of data consistency judgment. Therefore, in the synchronous verification process, this invention selects a portion of user behavior data from non-critical data to upgrade as verification objects based on the distribution density and index interval of critical data, in order to fill the verification gaps between critical data and improve the overall synchronous verification quality, since integrity verification is still required subsequently.

[0150] If the synchronization check fails, determine whether the primary key of the critical data can be mapped.

[0151] If the primary key of the key data can be mapped, the key data is collected from the collection database according to the mapped primary key.

[0152] If the primary key of critical data cannot be mapped, the control acquisition database randomly retransmits data packets to the verification database, and the verification database determines whether the retransmitted data packets are missing.

[0153] A primary key that can be mapped means that the corresponding user behavior record (such as a purchase behavior) can be directly located in the data collection database through a unique identifier. In this case, missing data can be collected directly from the source without being lost. If the primary key is not mappable, it means that the corresponding record cannot be found directly, and a coarser-grained method (such as retransmitting the entire data packet) must be used to fill in as much missing data as possible.

[0154] Get the number of retransmitted data packets In the retransmitted data packets, the number of times the database's missing critical data was checked was [number missing]. The number of missing non-keywords is ;

[0155] Calculate missing value , and Use weighting coefficients; set missing value thresholds. ;

[0156] The merged synchronization check table is sent to the main storage database.

[0157] when If so, the data packet will be sent to the buffer database.

[0158] When the synchronization verification result shows that critical data is missing, the present invention further performs hierarchical processing based on the mappability of the critical data primary key:

[0159] If the primary key of the key data can be mapped, the corresponding key data is directly obtained from the collection database according to the mapping relationship; if the primary key of the key data cannot be mapped, the collection database is controlled to retransmit the data packet in an attempt to complete the missing data.

[0160] The current data packet is temporarily stored in a buffer database determined based on the data synchronization link.

[0161] Obtain any database that does not participate in writing to the main storage database, and calculate the transmission time for establishing a data synchronization link between the database and the verification database and the processing database;

[0162] The sum of the transmission durations is denoted as the transmission cost;

[0163] The database with the lowest transmission cost is used as the buffer database, and the verification database exists. When this happens, the data packets currently stored in the verification database are transferred to the buffer database.

[0164] Due to network link fluctuations and the uncertainty of data status at the acquisition end, retransmitted data packets cannot be guaranteed to be complete. Therefore, a buffer database is introduced to temporarily store data packets that cannot be safely written to the main storage database at the moment. When subsequent retransmissions or data completion operations occur, the data packets in the buffer database can serve as a backup source, thereby reducing the number of repeated source calls and improving the overall data repair efficiency.

[0165] Example 2: In this example, the data management of real-time patient monitoring, remote surgery, and health monitoring devices is applied to medical and smart hospital scenarios. Data collected by different platforms or devices varies in quantity and importance. For example, critical data such as surgical operation records and ECG abnormalities are generated in real-time by surgical equipment and monitors, while non-critical data such as body temperature, blood pressure trends, and step count are collected periodically by bedside monitors or wearable devices. Furthermore, inconsistent network conditions across different devices may lead to data transmission delays or loss. By prioritizing real-time transmission, verification, and storage of critical data, it is ensured that critical medical information across platforms can be synchronized instantly and trigger alarms; while non-critical data can be delayed in storage or temporarily buffered, reducing the load under network fluctuations or high concurrency conditions. Through this approach, the timeliness and integrity of critical data are guaranteed in cross-device and cross-platform environments, while optimizing the processing of non-critical data, improving the overall reliability and response efficiency of the medical information system.

[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0167] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A cloud computing platform data storage method based on big data analysis, characterized in that: include: User behavior data is recorded in the database, and the user behavior data is evenly divided into data fragments and encrypted and encapsulated into data packets. The verification database and processing database each establish independent data synchronization links with the acquisition database to receive the data packets and send them to the main storage database after verification. Specifically: Estimate the data transmission rate of each data synchronization link; The processing database completes the full reception of the data packet before the verification database; The execution time cost verification strategy estimates the transmission time of data packets on the data synchronization link based on the data packet length, and calculates the synchronization verification time for the data packets in order to set two types of data transmission strategies; The two types of data transmission strategies are: one strategy in which the processing database sends data packets to the main storage database alone, and the other strategy in which the processing database and the verification database jointly send data packets to the main storage database. The single-path write-through verification strategy is executed, and the execution status of the two data transmission strategies is controlled based on the result of the main storage database's data packet integrity verification in one type of strategy. When executing a type II strategy: Implement database reconstruction strategies, jointly process and verify database data packets, upgrade non-critical data based on the distribution density of critical data, and reconstruct user behavior data records; The current data packet is temporarily stored in a buffer database determined based on the data synchronization link.

2. The cloud computing platform data storage method based on big data analysis according to claim 1, characterized in that: The estimation of the data transmission rate for each data synchronization link includes: For any data synchronization link: Set time window , obtain in Number of data packets attempted to be sent by the data synchronization link in the middle ; Get in Number of data packets successfully received by the data synchronization link ; Calculate the packet loss rate of the data synchronization link. ; Calculate retransmission coefficient , These represent the number of retransmissions and the number of packet losses within the time window, respectively. Each lost packet can correspond to one or more retransmissions. Estimate the data transmission rate of the data synchronization link. The data transmission rate is used to estimate the transmission time of data packets in the data synchronization link.

3. The data storage method for a cloud computing platform based on big data analytics according to claim 2, characterized in that: The execution time cost verification strategy estimates the transmission time of data packets on the data synchronization link based on the data packet length and calculates the synchronization verification time for the data packets, including: Obtain the length of the untransmitted link in the data synchronization link between the verification database and the acquisition database. ; Get the length of a single data packet ,calculate , The total length of the data packet; Calculate the time elapsed between the data packet acquisition database and the verification database. ; The data packet verification includes integrity verification and synchronization verification. The main storage database performs integrity verification on the data packets, while the processing database and the verification database perform synchronization verification on the data packets. Calculate the historical synchronization verification duration of the verification database. To retrieve any single data packet from the past; The data packet is parsed into records of individual user behavior data, the number of which is... ; For any record: Computing the record complexity , For the standard length of Bitcoin records, These are the weighting coefficients. Bitcoin length The parsing complexity of the data packets; Get the verification capabilities of the verification database Calculate the duration of synchronous verification. ; The user behavior data is divided into critical data and non-critical data. The average time for synchronous verification of critical data and non-critical data is calculated and denoted as follows: and ; Get the number of critical and non-critical data in the data packet being parsed. and ; calculate .

4. The data storage method for a cloud computing platform based on big data analytics according to claim 3, characterized in that: The above describes the setting of two types of data transmission strategies, including: The link length from the processing database to the main storage database is obtained. ; Calculate the transmission time of data packets sent from the processing database to the main storage database. ; like Configure the processing database to send data packets separately to the main storage database; like The configuration involves the processing database and the verification database synchronizing and verifying the data packets before sending them to the main storage database.

5. The data storage method for a cloud computing platform based on big data analytics according to claim 4, characterized in that: The single-path direct write verification strategy controls the execution status of two data transmission strategies based on the result of the main storage database's data packet integrity verification in one strategy, including: For any data packet obtained by the main storage database from the processing database, user behavior data, which is considered key data, is extracted from the data packet and substituted into the CRC function as a parameter to obtain the checksum field. ; Obtain the standard verification fields encapsulated in the data packet itself. Set the verification error threshold ; like If the integrity verification passes, the main storage database sends a shutdown synchronization verification process to the processing database. The verification database stops receiving data packets from the acquisition database and does not perform synchronization verification; like If the integrity verification fails, the data packet that fails will be recorded as an abnormal data packet. Obtain the link length from the data collection database to the processing database. Calculate the time to retrieve data from the source. ; like Then the main storage database will return the abnormal data packets in batches to the processing database, and the processing database will request replacement abnormal data packets from the collection database. If the attempt to replace the abnormal data packet at the source fails, then the second-class strategy will be executed. If the replacement of the abnormal data packet is successful, the replacement abnormal data packet will be sent to the main storage database.

6. The data storage method for a cloud computing platform based on big data analytics according to claim 3, characterized in that: The database reconstruction strategy involves jointly processing data packets from the database and verifying the database, and performing critical data upgrades on non-critical data based on the critical data distribution density, including: When the data packets received from the processing database and the verification database correspond to the same user behavior data and are critical data, synchronous verification is performed on the single piece of user behavior data: Retrieve records of single user behavior data from two sources, determine if the primary keys are the same, and compare the hash values ​​of the same fields in the records if the primary keys are the same; If the hash values ​​are the same, the synchronization verification passes. All user behavior data from this verification are merged to obtain a synchronization verification table, and the distribution density of key data is calculated. , For the amount of key data, The total number of user behavior data; Set key data density thresholds ; like Retrieve the index of any adjacent key data in the synchronization check table. ; Calculate the index interval ; Set the upper limit of the index interval ,like Then in key data and Filter out non-critical data and upgrade the non-critical data to critical data; The index for non-critical data used for upgrades is , Send the upgraded synchronization checklist to the main storage database.

7. The data storage method for a cloud computing platform based on big data analytics according to claim 6, characterized in that: The implementation of the database reconstruction strategy, which involves jointly processing data packets from the database and verifying the database, and performing critical data upgrades on non-critical data based on the critical data distribution density, also includes: If the synchronization check fails, determine whether the primary key of the critical data can be mapped. If the primary key of the key data can be mapped, the key data is collected from the collection database according to the mapped primary key. If the primary key of critical data cannot be mapped, the control acquisition database randomly retransmits data packets to the verification database, and the verification database determines whether the retransmitted data packets are missing. Get the number of retransmitted data packets In the retransmitted data packets, the number of times the database's missing critical data was checked was [number missing]. The number of missing non-keywords is ; Calculate missing value , and These are the weighting coefficients; Set missing threshold ; like The merged synchronization check table is sent to the main storage database. like If so, the data packet will be sent to the buffer database.

8. The data storage method for a cloud computing platform based on big data analytics according to claim 7, characterized in that: The step of transferring the current data packet to a buffer database determined according to the data synchronization link for temporary storage includes: Obtain any database that does not participate in writing to the main storage database, and calculate the transmission time for establishing a data synchronization link between the database and the verification database and the processing database; The sum of the transmission durations is denoted as the transmission cost; The database with the lowest transmission cost is used as the buffer database, and the verification database exists. When this happens, the data packets currently stored in the verification database are transferred to the buffer database.

Images