A cloud computing-based software data analysis method and system

Abstract

This invention discloses a software data analysis method and system based on cloud computing, belonging to the field of cloud computing technology. The method includes the following steps: collecting access records of data shards in a distributed file system, constructing a business-related access log, and calculating the access heat index and periodic feature vector of the data shards; extracting the target data shard identifier and matching the target data shard identifier with the periodic feature vector; scheduling a background thread to stream the prefetched target shard from the cold storage medium to the hot storage buffer, and adjusting the corresponding I / O scheduling priority according to the preheating state. This invention solves the problem in existing technologies where static cold / hot layered architectures cannot perceive sudden changes in business access patterns, leading to delayed cold data determination, cross-layer queries triggering data migration storms, and consequently, performance fluctuations in hot data queries. This is achieved by constructing a business-aware prediction-prefetch-feedback closed-loop mechanism.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a software data analysis method and system based on cloud computing. Background Technology

[0002] As enterprises deepen their digital transformation, the data analysis business carried by cloud computing platforms is becoming increasingly complex. When storing massive amounts of data in distributed file systems, a tiered storage strategy of hot and cold data is commonly adopted to balance performance and cost. Hot data is stored on high-performance media to ensure query speed, while cold data is stored on low-cost storage to control expenses.

[0003] However, in actual business operations, data access patterns are highly dynamic and unpredictable. For example, scenarios such as monthly financial settlement, sudden public opinion monitoring, or security audit tracing can lead to a large amount of historical cold data being accessed at high frequency in an instant.

[0004] In existing technologies, the classification of hot and cold data is mainly based on static marking of the last access time or the access frequency within a fixed time window. For example, data that has not been accessed for more than 30 days is marked as cold data and migrated to cold storage. When a sudden business query accesses this cold data, the system has to perform synchronous cross-layer data reading, that is, load the cold data from the low-speed storage medium to the hot storage before returning the query results.

[0005] This mechanism has the following problems: First, the cold and hot data determination mechanism is lagging and cannot detect impending sudden changes in access patterns, causing the system to only respond passively when a sudden query arrives; second, the cross-layer data loading process takes a long time, causing a sharp increase in query latency and triggering the so-called "data migration storm"; third, the cold data loading process will preempt I / O resources originally used to process hot data queries, causing hot data queries to also be blocked and queued, resulting in severe fluctuations in the response latency of the entire analysis platform, and in severe cases, even causing the system to freeze, making it difficult to meet the enterprise's requirements for the real-time performance and stability of data analysis. Summary of the Invention

[0006] This application provides a cloud computing-based software data analysis method and system, which solves the problem that the static cold and hot data layered architecture in the prior art cannot detect sudden changes in business access patterns, resulting in delayed cold data determination, cross-layer queries triggering data migration storms, and consequently causing resource contention and performance fluctuations in hot data queries.

[0007] This application provides a software data analysis method based on cloud computing, including: collecting access records of data shards in a distributed file system, extracting feature information including business identifiers, access timestamps and query contexts, and constructing business-related access logs; Based on business-related access logs, calculate the access popularity index and periodic feature vector of data shards, and divide the data shards into four logical states: deep archiving, low-frequency standby, high-frequency activity, and instantaneous burst. Parse the received software data analysis query statement, extract the target data segment identifier, match the target data segment identifier with the periodic feature vector, and determine the prefetch target segment that is in a low-frequency standby state and meets the preset prediction conditions. For each prefetch target fragment, a migration instruction is generated. The scheduling background thread streams the prefetch target fragment from the cold storage medium to the hot storage buffer and updates the metadata status of the prefetch target fragment to the preheating state. During the software data analysis and query execution phase, the prefetch target fragment in the hot storage buffer is directly called, and the logical state is updated according to the actual access results of the prefetch target fragment. At the same time, the corresponding I / O scheduling priority is adjusted according to the preheating state.

[0008] Furthermore, the steps for constructing business-related access logs include: Deploy a monitoring plugin on each storage node of the distributed file system; Extract multi-source fields from the parameter structure and combine them to define a business identifier; The business identifier is concatenated with the access timestamp to form a structured log record, which is in the format of a JSON object; All log records reported by storage nodes are merged according to data shard identifiers to construct a business-related access log table organized by data shard primary key.

[0009] Furthermore, the steps for calculating the visit popularity index include: Read the N most recent access records corresponding to a certain data shard in the business-related access log; When the window is full, remove the oldest record and insert the newest record; A time decay function is applied to each access record within the window to calculate a weight value for each record; the time decay function is defined as follows: ; in, This represents the weight value corresponding to the i-th access record within the window. It is the base of the natural logarithm. It is the attenuation coefficient, with a value range of (0,1]. The timestamp in milliseconds represents the current system time. This represents the value of the access_time field in the i-th access record. It is the normalized time constant; The access popularity index of the data shard is obtained by summing the weight values ​​of all records within the window. The calculation formula is: ; in, To measure the popularity index, The value is 1000.

[0010] Furthermore, the steps for calculating the periodic eigenvectors include: Extract all access records for each data shard over the past 30 days from the business-related access logs, aggregate the timestamps by hourly granularity, count the number of accesses within each hourly time period, and construct a time-series sequence. ,in This represents the number of visits within the k-th hour; For the time series Performing a Discrete Fourier Transform converts the time-domain signal into a frequency-domain signal. The calculation formula is as follows: ; in, Indicates the first Complex coefficients of each frequency component, The value is 720. It is a time series The One element, It is the imaginary unit, satisfying , Take 3.1415926, It is the base of the natural logarithm; Calculate the amplitude corresponding to each frequency component. The calculation formula is: ; in, Indicates the first The amplitude of each frequency component Representing complex numbers The modulus length, i.e. , The value is 720; Identify amplitude The local maxima points were selected, and the three frequency components with the largest amplitudes were chosen, corresponding to periods of . Hours, of which ; These three periods and their corresponding amplitude values ​​form a periodic eigenvector. The data is stored in the data shard metadata table of the metadata server.

[0011] Furthermore, the steps for determining the target prefetch fragment include: The received software data analysis query statements are subjected to lexical and syntactic analysis to generate an abstract syntax tree; Extract the time range conditions involved in the WHERE clause from the abstract syntax tree. If a time range condition exists, obtain the start time. and end time ; If no explicit time condition is specified, the historical query logs of the most recent week are associated with the business identifier in the query statement, and the most frequently accessed time window in the history of this type of query is taken as the estimated time window. The time window Periodic eigenvectors of data sharding Perform a match; for Each periodic component Calculate the peak access range corresponding to this period; Add the data shards marked as prefetch candidates to the list of target shards for prefetching, and submit the list to the prefetch scheduler.

[0012] Furthermore, scheduling background threads includes: After receiving the list of target shards to be prefetched, a request is sent to the cluster's monitoring service to obtain the global I / O load data of the current distributed file system. Calculate the currently available prefetch thread quota based on the I / O load data. The calculation formula is: ; in, The number of allocable prefetch threads. This is the maximum number of prefetch threads allowed by the system. It's disk utilization. This is the upper limit threshold for disk utilization. This is the current I / O queue depth. It is the upper limit threshold of the queue depth. Indicates rounding down; A prefetch task is created for each prefetch target slice, and these tasks are placed in a priority queue, with each task assigned a scheduling priority value. ; According to the calculated thread quota The appropriate number of prefetch tasks are retrieved from the priority queue and assigned to idle threads in the thread pool for execution.

[0013] Furthermore, the steps for updating the metadata state of the prefetch target shard include: When streaming begins, a metadata update request is sent to the metadata server; The metadata server uses a two-phase commit protocol to process the update request; After the prefetch task completes the transmission of all data blocks, it sends a commit confirmation message to the metadata server. After the metadata server completes the update, it writes a commit record to the temporary log and releases the metadata lock on the data shard. If a prefetch task fails or times out during transmission, the prefetch scheduler sends a rollback request to the metadata server. The metadata server deletes the pre-committed record from the temporary log and releases the data shard lock without making any changes to the storage path.

[0014] Furthermore, the steps for updating the logical state based on the actual access results include: When executing software data analysis queries, the system first requests data location information from the metadata server based on the data fragment identifiers involved in the query plan. The metadata server returns the current storage path and status flags; The statistics service maintains a delay counter, and updates the timestamp to the current time when a status change candidate message is received; The statistics service runs a status inspection task every minute to calculate idle time.

[0015] This application provides a cloud computing-based software data analysis system for implementing a cloud computing-based software data analysis method, including: The module includes a data acquisition module, a vector calculation module, a fragmentation determination module, a status update module, and a priority adjustment module. The record acquisition module is used to collect access records of data shards in the distributed file system, extract feature information including business identifiers, access timestamps and query contexts, and construct business-related access logs. The vector calculation module is used to calculate the access popularity index and periodic feature vector of data shards based on business-related access logs, and to divide the data shards into four logical states: deep archiving, low-frequency standby, high-frequency activity, and instantaneous burst. The segment determination module is used to parse the received software data analysis query statement, extract the target data segment identifier, match the target data segment identifier with the periodic feature vector, and determine the pre-fetch target segment that is in a low-frequency standby state and meets the preset prediction conditions. The state update module is used to generate migration instructions for the prefetch target fragment, schedule a background thread to stream the prefetch target fragment from the cold storage medium to the hot storage buffer, and update the metadata status of the prefetch target fragment to the preheating state. The priority adjustment module is used to directly call the prefetch target fragment in the hot storage buffer during the software data analysis and query execution phase, update the logical state according to the actual access results of the prefetch target fragment, and adjust the corresponding I / O scheduling priority according to the preheating state.

[0016] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: By constructing access logs containing business identifiers and calculating periodic feature vectors, the system gains the ability to predict data access trends from a business perspective, transforming the determination of hot and cold storage layers from lagging offline statistics to proactive awareness based on business context. By matching periodic feature vectors with query time windows in real time during the query parsing phase and dynamically determining the prefetch range based on the current business concurrency, deep coupling between prefetching decisions and business access intentions is achieved, avoiding resource waste caused by blind prefetching.

[0017] Furthermore, by introducing a preheating state and a corresponding I / O scheduling priority adjustment mechanism, a transition buffer is opened for prefetched data in hot storage, and it is given a differentiated service quality between cold and hot data. This ensures that prefetched data can obtain a better response speed than cold data when queried, and avoids misjudged data occupying hot storage resources for a long time.

[0018] This application, through the aforementioned business-aware prediction-prefetch-feedback closed-loop mechanism, smooths the rigid boundaries of hot and cold tiered storage, eliminates data migration storms and resource contention caused by cross-tier queries, and improves the performance stability and resource utilization of the cloud computing platform when handling mixed loads without increasing hardware investment. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a cloud computing-based software data analysis method provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of a cloud computing-based software data analysis system provided in an embodiment of this application. Detailed Implementation

[0020] This application provides a cloud computing-based software data analysis method and system, which solves the problems in the prior art where the static cold and hot data layered architecture cannot detect sudden changes in business access patterns, resulting in delayed cold data determination, cross-layer queries triggering data migration storms, and consequently causing resource contention and performance fluctuations in hot data queries.

[0021] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0022] like Figure 1 The diagram shown is a flowchart of a cloud-based software data analysis method provided in an embodiment of this application. The method is applied to a cloud-based software data analysis system and includes the following steps: collecting access records of data shards in a distributed file system, extracting feature information including business identifiers, access timestamps and query contexts, and constructing business-related access logs. Based on business-related access logs, calculate the access popularity index and periodic feature vector of data shards, and divide the data shards into four logical states: deep archiving, low-frequency standby, high-frequency activity, and instantaneous burst. Parse the received software data analysis query statement, extract the target data segment identifier, match the target data segment identifier with the periodic feature vector, and determine the pre-fetch target segment that is in a low-frequency standby state and meets the preset prediction conditions. For each prefetch target fragment, a migration instruction is generated. The scheduling background thread streams the prefetch target fragment from the cold storage medium to the hot storage buffer and updates the metadata status of the prefetch target fragment to the preheating state. During the software data analysis and query execution phase, the prefetch target fragment in the hot storage buffer is directly called, and the logical state is updated according to the actual access results of the prefetch target fragment. At the same time, the corresponding I / O scheduling priority is adjusted according to the preheating state.

[0023] Furthermore, the steps for constructing business-related access logs include: A monitoring plugin is deployed on each storage node of the distributed file system. The monitoring plugin captures every read() and write() system call for data shards by mounting the Linux kernel's Ftrace framework or eBPF technology. When a user-space process initiates a read or write request, the monitoring plugin intercepts the system call and reads the parameter structure of the call. The multi-source field combination extracted from the parameter structure is defined as a business identifier. The multi-source field combination defined as a business identifier includes: the user identity identifier UID of the process that initiated the call, the project number ProjectID of the software project to which the process belongs in the project management system, and the query feature code QueryHash obtained by hashing the predicate conditions in the structured query statement. The query feature code is generated by performing an MD5 operation on the string of the WHERE clause in the SQL statement and taking the first 16 bits as the QueryHash. The monitoring plugin obtains the current nanosecond-level time of the server hardware clock, formats it as a Unix timestamp in milliseconds, and retains the precision to the millisecond level. The business identifier is concatenated with the access timestamp to form a structured log record, which is in the format of a JSON object and contains the following fields: data_block_id (globally unique identifier for data block, string type), uid (user identifier, unsigned 32-bit integer), project_id (project number, unsigned 16-bit integer), query_hash (query feature code, string type), access_time (access timestamp, unsigned 64-bit integer, unit: milliseconds), and rw_type (read / write type, enumeration value "read" or "write"). All log records reported by storage nodes are merged according to the data shard identifier and written to the corresponding topic in the distributed message queue Kafka. The log consumption service then pulls them in batches and persists them to the HBase database, constructing a business-related access log table organized by the data shard primary key. The primary key of this table is data_block_id, and the above fields are stored under the column family "access_records".

[0024] Furthermore, the steps for calculating the visit popularity index include: Read the most recent N access records for a certain data shard in the business-related access log, where N is 1000. That is, each data shard maintains a sliding window of length 1000 in memory, and stores the most recent 1000 access records in descending order of timestamp within the window. When the window is full, remove the oldest record and insert the newest record; A time decay function is applied to each access record within the window to calculate a weight value for each record; the time decay function is defined as follows: ; in, This represents the weight value corresponding to the i-th access record within the window. It is the base of the natural logarithm. It is the attenuation coefficient, with a value range of (0,1], and is set to 0.1 in this embodiment; The timestamp in milliseconds represents the current system time. This represents the value of the access_time field in the i-th access record. It is the normalized time constant, with a value of 3,600,000 milliseconds (i.e., 1 hour), used to normalize time differences to the hour level; The access popularity index of the data shard is obtained by summing the weight values ​​of all records within the window. The calculation formula is: ; in, To measure the popularity index, With a value of 1000, this index reflects the overall access popularity of data shards in a recent period, with accesses closer to the current time contributing more.

[0025] Furthermore, the steps for calculating the periodic eigenvectors include: Extract all access records for each data shard over the past 30 calendar days from the business-related access logs, aggregate the timestamps at the hourly granularity, count the number of accesses within each hourly time period, and construct a time series of length 720 (30 days × 24 hours). ,in This represents the number of visits within the k-th hour; For the time series Performing a Discrete Fourier Transform converts the time-domain signal into a frequency-domain signal. The calculation formula is as follows: ; in, Indicates the first The complex coefficients of each frequency component contain amplitude and phase information. The value is 720. It is a time series The One element, It is the imaginary unit, satisfying , Take 3.1415926, It is the base of the natural logarithm; Calculate the amplitude corresponding to each frequency component. The calculation formula is: ; in, Indicates the first The amplitude of each frequency component Representing complex numbers The modulus length, i.e. , The value is 720; Identify amplitude The local maxima points were selected, and the three frequency components with the largest amplitudes were chosen, corresponding to periods of . Hours, of which ; These three periods and their corresponding amplitude values ​​form a periodic eigenvector. The data is stored in the data shard metadata table of the metadata server; This vector is used to describe the main access cycle pattern of data shards. For example, T=24 hours corresponds to a daily cycle, and T=168 hours corresponds to a weekly cycle.

[0026] Furthermore, the steps for determining the target prefetch fragment include: The received software data analysis query statements are subjected to lexical and syntactic analysis to generate an abstract syntax tree; Extract the time range conditions involved in the WHERE clause from the abstract syntax tree. If a time range condition exists, obtain the start time. and end time ; If no explicit time condition is specified, the historical query logs of the most recent week are associated with the business identifier in the query statement, and the most frequently accessed time window in the history of this type of query is taken as the estimated time window. The time window Periodic eigenvectors of data sharding Perform a match; for Each periodic component Calculate the peak access range corresponding to this period; Specifically, by analyzing the consecutive time periods with the highest access volume within each period of historical data, a mapping relationship is established: for each period The peak access period is within 24 hours. Hours, of which This indicates the offset time of the peak access occurrence within this period. Indicates the half-width of the interval. and The units are all hours, and the values ​​are derived from historical statistics; Determine the time window Does it intersect with any peak access interval? The determination method is: and Convert the offset within the period to hours, i.e., calculate and We obtained two remainders. and The unit is milliseconds; if the interval and interval If there is overlap, it is determined that there is an intersection; For data shards that overlap and are currently in a low-frequency standby state, further obtain the business identifier corresponding to the data shard, and query the running query monitoring table to find the number of queries currently being executed that have the same project_id and query_hash prefix, and record this as the concurrency count. ; If the number of concurrent users Exceeding the preset concurrency threshold , If the value is 5, all data fragments in the low-frequency standby state associated with this business identifier will be marked as prefetch candidates; If the number of concurrent users If the threshold is not exceeded, only the data segments that are determined to have intersection are marked as prefetch candidates; Add the data shards marked as prefetch candidates to the list of target prefetch shards, and submit the list to the prefetch scheduler.

[0027] Furthermore, scheduling background threads includes: After receiving the list of target shards for prefetching via the prefetch scheduler, a request is sent to the cluster's monitoring service to obtain the global I / O load data of the current distributed file system; the I / O load data includes the disk utilization rate of each storage node. (Value range 0~100%) Current I / O queue depth and the average I / O throughput over the most recent minute ; Calculate the currently available prefetch thread quota based on the I / O load data. The calculation formula is: ; in, The number of allocable prefetch threads, rounded to the nearest integer; This is the maximum number of prefetch threads allowed by the system, with a value of 10. It's disk utilization. This is the upper limit threshold for disk utilization, with a value of 80%. This is the current I / O queue depth. This is the upper limit threshold for queue depth, with a value of 32. Indicates rounding down; The prefetch scheduler creates a prefetch task for each prefetch target slice and places these tasks into a priority queue, assigning each task a scheduling priority value. Its value ranges from 0 to 7, and the smaller the value, the higher the priority. Priority of prefetching tasks The priority is fixed at 5. This priority is lower than the I / O priority of front-end user query requests (front-end query priority is set to 2), but higher than the I / O priority of back-end data archiving tasks (archiving task priority is set to 7). According to the calculated thread quota The appropriate number of prefetched tasks are retrieved from the priority queue and assigned to idle threads in the thread pool for execution; the thread pool size is dynamically adjusted, but does not exceed [a certain value]. ; Each execution thread invokes the operating system's asynchronous I / O interface (such as io_uring or libaio) to initiate a data read request, reading data blocks of the target shard from cold storage media (such as AWS S3 or HDFS cold storage nodes); the read data blocks are temporarily stored in the thread's local buffer, and a write operation is triggered every 64MB read to write the data to a specified path in high-performance hot storage (such as a distributed cache layer composed of SSDs); the entire transmission process adopts a pipelined approach to ensure that read and write operations overlap and minimize transmission latency.

[0028] Furthermore, the steps for updating the metadata state of the prefetch target shard include: When streaming begins, a metadata update request is sent to the metadata server. The request includes: data block identifier data_block_id, original cold storage path cold_path (e.g., "s3: / / cold-bucket / data / block_123.parquet"), target hot storage path hot_path (e.g., " / hot_cache / block_123.parquet"), and transaction identifier trans_id for this operation. The metadata server uses a two-phase commit protocol to process the update request. In the first phase, the metadata server locks the metadata record of the data shard to prevent concurrent updates and writes a pre-commit record to the temporary log, which includes: data_block_id, cold_path, hot_path, trans_id, and the status flag "preparing". After the prefetch task completes the transmission of all data blocks, it sends a commit confirmation message to the metadata server. After receiving the confirmation, the metadata server updates the storage path field in the metadata record from cold_path to hot_path, and adds a new status field state, whose value is set to "prewarming". This field is an enumeration value, and the valid values ​​are "cold", "prewarming", "hot" and "burst". After the metadata server completes the update, a commit record is written to the temporary log, the status flag is updated to "committed", and the metadata lock of the data shard is released. At the same time, a special flag is set for data fragments in the "prewarming" state, instructing the cache management module not to include the space occupied by the fragment when calculating the hot storage cache quota. That is, the fragment is allowed to temporarily exceed the cache capacity limit, but the exceedance limit shall not exceed 10% of the total cache capacity. If a prefetch task fails or times out during transmission (not completed within 300 seconds), the prefetch scheduler sends a rollback request to the metadata server. The metadata server deletes the pre-committed record from the temporary log and releases the data shard lock without making any changes to the storage path.

[0029] Furthermore, the steps for updating the logical state based on the actual access results include: When executing software data analysis queries, the system first requests data location information from the metadata server based on the data fragment identifiers involved in the query plan. The metadata server returns the current storage path and status flags; If the returned status is marked as "prewarming", the query engine reads the data directly from the hot storage buffer path and sends a status change candidate message to the statistics service, carrying the fields: data_block_id and access_time; The statistics service maintains a latency counter, which records the timestamp of the last time a data shard in the "prewarming" state was actually queried. When a status change candidate message is received, update the timestamp to the current time. The statistics service runs a status inspection task every minute, scanning all data shards in the "prewarming" state; for each shard, it retrieves its current time. and last visit time Calculate idle time ; like Less than the preset effective heating period threshold , If the value is 3600000 milliseconds (1 hour), the logical status of the shard will be changed from "prewarming" to "hot" (high-frequency active status), and the metadata server will be notified to update the status field to "hot". At the same time, the cache management module will start to include the shard in the hot storage cache quota and participate in the normal LRU eviction policy. like Greater than or equal to If the fetch is misjudged, the migration mechanism is triggered. The statistics service sends a cleanup command to the cache management module, which then deletes the fragment from the hot storage buffer to release storage space. After deletion, the metadata server is notified to restore the status field to "cold" (low-frequency standby state) and restore the storage path to the original cold storage path.

[0030] like Figure 2The diagram shown is a structural schematic of a cloud computing-based software data analysis system provided in an embodiment of this application. The cloud computing-based software data analysis system provided in this embodiment includes: a record acquisition module, a vector calculation module, a fragmentation determination module, a status update module, and a priority adjustment module. The record acquisition module is used to collect access records of data shards in the distributed file system, extract feature information including business identifiers, access timestamps and query contexts, and construct business-related access logs. The vector calculation module is used to calculate the access popularity index and periodic feature vector of data shards based on business-related access logs, and to divide the data shards into four logical states: deep archiving, low-frequency standby, high-frequency activity, and instantaneous burst. The segment determination module is used to parse the received software data analysis query statement, extract the target data segment identifier, match the target data segment identifier with the periodic feature vector, and determine the pre-fetch target segment that is in a low-frequency standby state and meets the preset prediction conditions. The state update module is used to generate migration instructions for the prefetch target fragment, schedule a background thread to stream the prefetch target fragment from the cold storage medium to the hot storage buffer, and update the metadata status of the prefetch target fragment to the preheating state. The priority adjustment module is used to directly call the prefetch target fragment in the hot storage buffer during the software data analysis and query execution phase, update the logical state according to the actual access results of the prefetch target fragment, and adjust the corresponding I / O scheduling priority according to the preheating state.

[0031] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0032] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0033] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0034] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0035] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0036] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A software data analysis method based on cloud computing, characterized in that, Includes the following steps: Collect access records of data shards in the distributed file system, extract feature information including business identifiers, access timestamps and query context, and construct business-related access logs; Based on business-related access logs, calculate the access popularity index and periodic feature vector of data shards, and divide the data shards into four logical states: deep archiving, low-frequency standby, high-frequency activity, and instantaneous burst. Parse the received software data analysis query statement, extract the target data segment identifier, match the target data segment identifier with the periodic feature vector, and determine the prefetch target segment that is in a low-frequency standby state and meets the preset prediction conditions. For each prefetch target fragment, a migration instruction is generated. The scheduling background thread streams the prefetch target fragment from the cold storage medium to the hot storage buffer and updates the metadata status of the prefetch target fragment to the preheating state. During the software data analysis and query execution phase, the prefetch target fragment in the hot storage buffer is directly called, and the logical state is updated according to the actual access results of the prefetch target fragment. At the same time, the corresponding I / O scheduling priority is adjusted according to the preheating state.

2. The software data analysis method based on cloud computing as described in claim 1, characterized in that, The steps to build business-related access logs include: Deploy a monitoring plugin on each storage node of the distributed file system; Extract multi-source fields from the parameter structure and combine them to define a business identifier; The business identifier is concatenated with the access timestamp to form a structured log record, which is in the format of a JSON object; All log records reported by storage nodes are merged according to the data shard identifier to construct a business-related access log table organized by the data shard primary key.

3. The software data analysis method based on cloud computing as described in claim 1, characterized in that, The steps for calculating the popularity index include: Read the N most recent access records corresponding to a certain data shard in the business-related access log; When the window is full, remove the oldest record and insert the newest record; A time decay function is applied to each access record within the window to calculate a weight value for each record; the time decay function is defined as follows: ； in, This represents the weight value corresponding to the i-th access record within the window. It is the base of the natural logarithm. It is the attenuation coefficient, with a value range of (0,1]. The timestamp in milliseconds represents the current system time. This represents the value of the access_time field in the i-th access record. It is the normalized time constant; The access popularity index of the data shard is obtained by summing the weight values ​​of all records within the window. The calculation formula is: ； in, To measure the popularity index, The value is 1000.

4. The software data analysis method based on cloud computing as described in claim 1, characterized in that, The steps for calculating periodic eigenvectors include: Extract all access records for each data shard over the past 30 days from the business-related access logs, aggregate the timestamps by hourly granularity, count the number of accesses within each hourly time period, and construct a time-series sequence. ,in This represents the number of visits within the k-th hour; For the time series Performing a Discrete Fourier Transform converts the time-domain signal into a frequency-domain signal. The calculation formula is as follows: ； in, Indicates the first Complex coefficients of each frequency component, The value is 720. It is a time series The One element, It is the imaginary unit, satisfying , Take 3.1415926, It is the base of the natural logarithm; Calculate the amplitude corresponding to each frequency component. The calculation formula is: ； in, Indicates the first The amplitude of each frequency component Representing complex numbers The modulus length, i.e. , The value is 720; Identify amplitude The local maxima points were selected, and the three frequency components with the largest amplitudes were chosen, corresponding to periods of . Hours, of which ; These three periods and their corresponding amplitude values ​​form a periodic eigenvector. The data is stored in the data shard metadata table of the metadata server.

5. The software data analysis method based on cloud computing as described in claim 1, characterized in that, The steps to determine the target prefetch fragment include: The received software data analysis query statements are subjected to lexical and syntactic analysis to generate an abstract syntax tree; Extract the time range conditions involved in the WHERE clause from the abstract syntax tree. If a time range condition exists, obtain the start time. and end time ; If no explicit time condition is specified, the historical query logs of the most recent week are associated with the business identifier in the query statement, and the most frequently accessed time window in the history of this type of query is taken as the estimated time window. The time window Periodic eigenvectors of data sharding Perform a match; for Each periodic component Calculate the peak access interval corresponding to this period; Add the data shards marked as prefetch candidates to the list of target prefetch shards, and submit the list to the prefetch scheduler.

6. The software data analysis method based on cloud computing as described in claim 1, characterized in that, Scheduling background threads includes: After receiving the list of target shards to be prefetched, a request is sent to the cluster's monitoring service to obtain the global I / O load data of the current distributed file system. Calculate the currently available prefetch thread quota based on the I / O load data. The calculation formula is: ； in, The number of allocable prefetch threads. This is the maximum number of prefetch threads allowed by the system. It's disk utilization. This is the upper limit threshold for disk utilization. This is the current I / O queue depth. It is the upper limit threshold of the queue depth. Indicates rounding down; A prefetch task is created for each prefetch target slice, and these tasks are placed in a priority queue, with each task assigned a scheduling priority value. ; According to the calculated thread quota The appropriate number of prefetch tasks are retrieved from the priority queue and assigned to idle threads in the thread pool for execution.

7. The software data analysis method based on cloud computing as described in claim 1, characterized in that, The steps for updating the metadata status of the prefetch target shard include: When streaming begins, a metadata update request is sent to the metadata server; The metadata server uses a two-phase commit protocol to process the update request; After the prefetch task completes the transmission of all data blocks, it sends a commit confirmation message to the metadata server. After the metadata server completes the update, it writes a commit record to the temporary log and releases the metadata lock on the data shard. If a prefetch task fails or times out during transmission, the prefetch scheduler sends a rollback request to the metadata server. The metadata server deletes the pre-committed record from the temporary log and releases the data shard lock without making any changes to the storage path.

8. The software data analysis method based on cloud computing as described in claim 1, characterized in that, The steps for updating the logical state based on the actual access results include: When executing software data analysis queries, the system first requests data location information from the metadata server based on the data fragment identifiers involved in the query plan. The metadata server returns the current storage path and status flags; The statistics service maintains a delay counter, and updates the timestamp to the current time when a status change candidate message is received; The statistics service runs a status inspection task every minute to calculate idle time.

9. A cloud computing-based software data analysis system, used to implement the cloud computing-based software data analysis method according to any one of claims 1-8, characterized in that, include: The module includes a data acquisition module, a vector calculation module, a fragmentation determination module, a status update module, and a priority adjustment module. The record acquisition module is used to collect access records of data shards in the distributed file system, extract feature information including business identifiers, access timestamps and query contexts, and construct business-related access logs. The vector calculation module is used to calculate the access popularity index and periodic feature vector of data shards based on business-related access logs, and to divide the data shards into four logical states: deep archiving, low-frequency standby, high-frequency activity, and instantaneous burst. The segment determination module is used to parse the received software data analysis query statement, extract the target data segment identifier, match the target data segment identifier with the periodic feature vector, and determine the pre-fetch target segment that is in a low-frequency standby state and meets the preset prediction conditions. The state update module is used to generate migration instructions for the prefetch target fragment, schedule a background thread to stream the prefetch target fragment from the cold storage medium to the hot storage buffer, and update the metadata status of the prefetch target fragment to the preheating state. The priority adjustment module is used to directly call the prefetch target fragment in the hot storage buffer during the software data analysis and query execution phase, update the logical state according to the actual access results of the prefetch target fragment, and adjust the corresponding I / O scheduling priority according to the preheating state.

Images