Cloud-computing-based software development data hierarchical processing method and system

Abstract

The application is specifically a software development data hierarchical processing method and system based on cloud computing, and relates to the technical field of cloud computing and distributed storage, comprising: collecting access intention events related to software development operations from a cloud layer, an edge layer and a terminal layer respectively, and encapsulating the events collected from each layer in a unified event format. In the application, multi-source streaming access intention events are collected from the cloud layer, the edge layer and the terminal layer, and an intention strength index is calculated based on a sliding window aggregation, so as to solve the technical problem that a static hierarchical strategy cannot perceive real-time data access requirements; a migration instruction is generated by comparing an edge layer missing penalty value with a storage occupancy cost, and data eviction is performed in combination with a survival time threshold and a global access frequency decay ratio, so as to solve the problem that the cache space of the edge layer is long-term occupied by low-frequency data.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing and distributed storage technology, and in particular to a method and system for layered data processing in software development based on cloud computing. Background Technology

[0002] In large-scale distributed software development scenarios, the software development process involves various data types such as source code version repositories, compilation intermediate artifacts, dependency packages, test case sets, and container images. These data are distributed and stored in three layers: the central code repository in the cloud layer, the CI / CD build nodes in the edge layer, and the developer devices in the terminal layer.

[0003] Existing data tiering management solutions mostly adopt a centralized static tiering strategy, that is, data is divided into different storage tiers according to preset fixed rules; However, the static layering strategy also has areas that need optimization: First, the inability to perceive developers' real-time changing data access needs leads to hot data not being promptly pushed down to the low-latency storage layer, causing the edge layer and terminal layer to frequently initiate data retrieval requests to the cloud, increasing access latency and network bandwidth overhead. Second, the cache management at each level remains independent of each other, lacking cross-level collaborative scheduling capabilities. It cannot use the fine-grained operation context of the terminal layer to guide the edge layer cache preheating, nor can it use the global topology information of the cloud to optimize the edge layer data eviction decision. This results in low utilization of high-speed storage space in the edge layer and the cache hit rate is difficult to meet the requirements of continuous integration pipelines.

[0004] Therefore, a data layering processing method and system for software development based on cloud computing is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a cloud computing-based data layering processing method and system for software development, in order to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: Cloud-based software development data tiering methods include: The configuration information of each node in the cloud layer, edge layer, and terminal layer is obtained, and a global data routing table is generated based on the configuration information. The global data routing table records the routing entry for each data block. Each routing entry includes a unique identifier for the data block, a hash value of the data block content, a current storage location mapping, and a policy version number. Access intent events related to software development operations are collected from the cloud layer, edge layer, and terminal layer respectively. The events collected from each layer are encapsulated in a unified event format to form a multi-source access intent event stream. The multi-source access intent event stream is aggregated using a sliding window. Within each sliding window, for each data block involved, feature values ​​of three dimensions—edit frequency, cumulative hover duration, and construction context matching degree—are extracted. After normalizing the feature values, they are weighted and summed to calculate the intent intensity index of each data block. For each data block, the global data routing table is queried to obtain the current residence level. The intent intensity index is compared with a preset high intensity threshold. When the intent intensity index is greater than the high intensity threshold, a data sinking instruction or a terminal pre-push instruction is generated by combining the normalized comparison result of the edge layer missing penalty value and the edge layer storage occupation cost. When the intent intensity index is not greater than the high intensity threshold, a data eviction instruction is generated by combining the continuous residence time and the global access frequency attenuation ratio, forming a set of hierarchical migration instructions. The hierarchical migration instruction set is read, a shadow file of the data block is created on the target node, and the integrity of the content hash value is verified. After the verification is passed, the storage location mapping of the global data routing table is updated atomically, and the shadow file is converted into a formal data file.

[0007] Preferably, the global data routing table is persistently stored in the cloud layer, and an initial snapshot of the global data routing table is distributed to each edge layer node and terminal layer device. The policy version number is a monotonically increasing integer value. After each layer migration decision is completed and the global data routing table is updated, the cloud layer increments the policy version number by one, generates an incremental change to the global data routing table containing the new policy version number, and pushes it to each edge layer node and terminal layer device. After receiving the incremental change to the global data routing table, the edge layer nodes and terminal layer devices compare the version number with the locally cached global data routing table. Only when the received policy version number is greater than the local version number will the incremental change be applied to the local copy of the global data routing table.

[0008] Preferably, the collection of access intent events related to software development operations from the cloud layer, edge layer, and terminal layer respectively includes: At the terminal layer, the developer's operation events are captured through plug-in programs deployed in the integrated development environment. These operation events include cursor hovering over code symbol names, Git branch switching events, and unit test start events. Each operation event carries a session identifier, a timestamp, and the name of the operation object. At the edge layer, pipeline trigger events and dependency description file parsing events are captured by parsing the CI / CD pipeline configuration file. Each event carries a build identifier, code commit hash value, and the name of the dependency involved. At the cloud layer, event notifications from the code repository are received through the API gateway, capturing merge request creation events and version tag creation events. Each event carries the repository identifier, source branch name, and target branch name. The unified event format includes event source level, event type identifier, timestamp, list of associated data block identifiers, and semantic attribute fields. The list of associated data block identifiers is generated by querying the global data routing table to obtain the unique identifier of the corresponding data block based on the operation object name or dependency name carried in the event.

[0009] Preferably, before encapsulating the events according to a unified event format, noise filtering is performed on the raw events collected by the terminal layer. The noise filtering includes: for each terminal session, within a continuous time window, counting the triggering frequency of the same type of events on the same operation object; when the triggering frequency exceeds a preset upper limit threshold and the operation object between events has not changed, merging the repeated events within the time window and retaining only the last event as a valid event for output to the subsequent processing flow.

[0010] Preferably, the calculation of the intent intensity index for each data block includes: Within each sliding window, the editing frequency and cumulative hover duration are normalized to the mean based on the range, scaling them to the range of zero to one. The construction context matching degree is determined by judging whether the data block belongs to the set of dependency names declared in the current active CI / CD pipeline construction configuration file. If it belongs, the value is one; if it does not belong, the value is zero. When multiple active pipelines exist at the same time, the maximum value of the matching degree value corresponding to each pipeline is taken. The intent intensity index is equal to the sum of the product of the edit frequency weight coefficient and the normalized edit frequency value, the product of the hover duration weight coefficient and the normalized hover cumulative duration value, and the product of the construction context matching degree weight coefficient and the construction context matching degree value, wherein the sum of the edit frequency weight coefficient, the hover duration weight coefficient, and the construction context matching degree weight coefficient is equal to one. The intention intensity index calculated by the current sliding window is subjected to an exponentially weighted moving average with the smoothed intention intensity index output by the previous sliding window to obtain the smoothed intention intensity index of the current sliding window. The smoothed intention intensity index is equal to the product of the smoothing coefficient and the original intention intensity index of the current sliding window plus a subtraction of the product of the smoothing coefficient and the smoothed intention intensity index of the previous sliding window.

[0011] Preferably, generating a data sinking instruction or a terminal pre-push instruction when the intent intensity index is greater than the high intensity threshold includes: Calculate the edge layer missing penalty value, which is equal to the product of the intent strength index of the data block, the estimated transmission delay of downloading the data block from the cloud layer to the edge layer, and the estimated number of accesses to the data block within the current sliding window; Calculate the edge layer storage occupancy cost, which is equal to the ratio of the storage space size of the data block to the remaining high-speed storage capacity available to the current edge layer node; The edge layer missing penalty value and the edge layer storage cost are compared after being normalized by mean based on the range. If the normalized edge layer missing penalty value is greater than the normalized edge layer storage cost and the data block is currently residing at the cloud layer, a data sinking instruction is generated to migrate the data block from the cloud layer to the edge layer node that initiated the access request. If the normalized edge layer missing penalty value is greater than the normalized edge layer storage cost and the data block is currently residing at the edge layer, and the terminal layer issues a deep working signal for the code module to which the data block belongs, a terminal pre-push instruction is generated to migrate the data block from the edge layer to the local cache of the corresponding terminal layer device. The conditions for generating the depth working signal are: within multiple consecutive sliding windows, the frequency of editing operation events generated by the terminal layer for the data block set associated with the same code module is continuously higher than a preset depth working frequency threshold.

[0012] Preferably, generating a data eviction instruction when the intent intensity index is not greater than the high intensity threshold includes: The system obtains the continuous residence time of the data block at the current residence level and the global access frequency attenuation ratio. The global access frequency attenuation ratio is the ratio of the recent global access frequency of the data block to the historical peak global access frequency. The recent global access frequency is the total number of access requests generated by each edge layer node and terminal layer device for the data block in the current statistical period. The historical peak global access frequency is the historical maximum value of the total number of access requests in each statistical period since the data block was first included in the hierarchical management. When the historical peak count is zero, the global access frequency attenuation ratio is zero. When the continuous dwell time exceeds a preset lifetime threshold and the global access frequency attenuation ratio is lower than a preset attenuation threshold, a data eviction command is generated to migrate the data block from the current edge layer node to the cloud layer cold storage area.

[0013] Preferably, after generating the data sinking instruction, the process further includes dependency association sinking processing: obtaining the dependency association graph of the data block, which is generated by parsing the dependency description file of the software project and represents the dependency relationship between each data block in a directed acyclic graph structure; traversing the direct dependencies of the data blocks in the dependency association graph, and for the direct dependencies currently residing in the cloud layer, generating a related sinking instruction for the direct dependency and appending it to the hierarchical migration instruction set of the current decision cycle.

[0014] Preferably, after performing cross-layer migration of data blocks and updating the global data routing table, a streaming negative feedback correction step is further included: At the end of each statistical period, the edge layer counts the number of invalid preheating and cache penetration. The number of invalid preheating is the total number of data blocks that are sinked to the edge layer or pre-pushed to the terminal layer due to the hierarchical migration decision in the current statistical period but have zero actual access counts. The number of cache penetration is the total number of data blocks that reside in the cloud layer in the current statistical period and are not predicted to be in high access demand but have more than a preset access count threshold. The edge layer employs the FTRL online learning algorithm, using the number of invalid warm-ups and the number of cache penetrations as input loss feedback signals to incrementally update the editing frequency weight coefficient, hover duration weight coefficient, and construction context matching degree weight coefficient in the intent intensity index calculation.

[0015] A cloud-based software development data tiering processing system includes: The global routing initialization module is used to obtain the configuration information of each node in the cloud layer, edge layer and terminal layer, and generate a global data routing table based on the configuration information. The global data routing table records the routing entry for each data block. Each routing entry includes a unique identifier for the data block, a hash value of the data block content, a current storage location mapping and a policy version number. The multi-source event acquisition module is used to collect access intent events related to software development operations from the cloud layer, edge layer, and terminal layer respectively, and encapsulate the events collected at each layer according to a unified event format to form a multi-source access intent event stream. The intent intensity calculation module is used to perform sliding window aggregation processing on the multi-source access intent event stream. Within each sliding window, for each data block involved, it extracts feature values ​​of three dimensions: editing frequency, cumulative hover duration, and construction context matching degree. After normalizing the feature values, it performs weighted summation to calculate the intent intensity index of each data block. The migration decision generation module is used to query the global data routing table to obtain the current residence level for each data block, compare the intent intensity index with the preset high intensity threshold, and generate a data sinking instruction or terminal pre-push instruction by combining the normalized comparison result of the edge layer missing penalty value and the edge layer storage occupation cost when the intent intensity index is greater than the high intensity threshold. When the intent intensity index is not greater than the high intensity threshold, generate a data eviction instruction by combining the continuous residence time and the global access frequency attenuation ratio, forming a hierarchical migration instruction set. The migration execution module is used to read the hierarchical migration instruction set, create a shadow file of the data block on the target node and perform content hash value integrity verification. After the verification is passed, it atomically updates the storage location mapping of the global data routing table and converts the shadow file into a formal data file.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention addresses the technical problem that static layering strategies cannot perceive real-time data access needs by collecting multi-source streaming access intent events from the cloud layer, edge layer, and terminal layer and calculating the intent intensity index based on a sliding window aggregation. It also addresses the problem that edge layer cache space is occupied by low-frequency data for a long time by generating migration instructions based on the comparison between edge layer missing penalty value and storage occupation cost, and performing data eviction by combining lifetime threshold and global access frequency decay ratio.

[0017] 2. This invention performs cross-layer migration by writing shadow files and updating atomic global data routing tables, addressing the problem of abnormal data reading caused by intermediate migration states; it constructs a streaming negative feedback correction loop by updating the weights of the edge layer FTRL online learning algorithm and adjusting the global threshold in the cloud layer, addressing the technical problem of accumulated prediction deviations caused by the evolution of development activity modes as project stages, and achieving the technical effect that the hierarchical decision parameters can be continuously and adaptively adjusted. Attached Figure Description

[0018] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0020] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0021] Example 1

[0022] Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0023] In this embodiment, it includes: Step 1: Obtain the metadata of the three-tier architecture nodes and initialize the global data routing table. The system acquires configuration information for each node in the cloud, edge, and terminal layers. This configuration information includes the storage capacity, storage medium type, network address, and layer identifier for each node. A global data routing table is generated based on this configuration information. The global data routing table records a routing entry for each data block. Each routing entry includes a unique identifier for the data block, a hash value of the data block content, a current storage location mapping, and a policy version number. The current storage location mapping includes the resource locator of the cloud layer where the data block resides, the list of network addresses of the edge node, and the physical address of the terminal device. The policy version number is a monotonically increasing integer value used to identify the version of the current layered policy.

[0024] The global data routing table is persistently stored in the cloud layer, while initial snapshots of the global data routing table are distributed to each edge layer node and terminal layer device.

[0025] The aforementioned data blocks are independently addressable data units generated during software development. Specifically, they include a collection of source code files for a specific version, dependency package files for a specific version, a directory of intermediate compilation artifacts, test case sets, and container image layers. Each data block is identified by performing a SHA256 hash operation on its content, i.e., it is identified using content addressing.

[0026] Step 2: Collect multi-source streaming access intent events Access intent events related to software development operations are collected from the cloud layer, edge layer, and terminal layer respectively to obtain a multi-source access intent event stream.

[0027] At the terminal layer, the developer's operation events are captured by a plug-in program deployed in the integrated development environment. These operation events include cursor hover over code symbol name events, Git branch switching events, and unit test start events. Each operation event carries a session identifier, a timestamp, and the name of the operation object.

[0028] At the edge layer, pipeline trigger events and dependency description file parsing events are captured by parsing the CI / CD pipeline configuration file. Each event carries a build identifier, a code commit hash, and the name of the dependency involved.

[0029] At the cloud layer, event notifications from the code repository are received through the API gateway, capturing merge request creation events and version tag creation events. Each event carries the repository identifier, source branch name, and target branch name.

[0030] The events collected at each of the above levels are encapsulated in a unified event format, which includes the event source level, event type identifier, timestamp, list of associated data block identifiers, and semantic attribute fields, forming a unified format multi-source access intent event stream.

[0031] The list of associated data block identifiers is generated as follows: Based on the name of the operation object or dependency carried in the event, the global data routing table is queried to obtain the unique identifier of the data block corresponding to the operation object or dependency, and then filled into the associated data block identifier list. For example, when the terminal layer captures an event where the cursor hovers over a function name, the code index is queried based on the function name to determine the source code file where the function is located, and then the global data routing table is queried based on the source code file name to obtain the unique identifier of the corresponding data block.

[0032] To filter out invalid high-frequency input / output events generated by operations such as automatic saving in the integrated development environment and background index updates, noise filtering is performed on the raw events collected at the terminal layer before event encapsulation. The noise filtering method is as follows: for each terminal session, within a continuous time window, the triggering frequency of the same type of event on the same operation object is counted. When the triggering frequency exceeds a preset upper limit threshold and the operation object between events has not changed, the repeated events within that time window are merged, and only the last event is retained as a valid event for output to the subsequent processing flow.

[0033] Step 3: Perform sliding window aggregation on the access intent event stream and calculate the intent strength index for each data block. Based on a stream processing framework, a sliding window aggregation process is performed on multi-source access intent event streams. The window length and sliding step size of the sliding window are set. Within each sliding window, for each data block involved in the sliding window, three feature values ​​are extracted: the editing frequency, the cumulative hover duration, and the construction context matching degree of that data block within the current sliding window.

[0034] Due to editing frequency The units are times / second and cumulative hovering time. The unit is seconds, and the construction context matching degree The values ​​are dimensionless, taking the values ​​of 0 or 1. Since the dimensions of the three values ​​are inconsistent, a direct weighted summation cannot be performed. Therefore, before calculating the intention intensity index, [the following is necessary:] ... and Each sample is scaled to the mean using mean normalization based on the range. The interval eliminates the influence of dimensional differences on weighted calculations. Normalization is performed independently within each sliding window based on the distribution of feature values ​​corresponding to all data blocks within that window.

[0035] For each data block The intent intensity index within the current sliding window is calculated using the following formula. : ; in, The first one in the current sliding window One data block, For data blocks The intent intensity index within the current sliding window. For data blocks The normalized value of the frequency of edit operation events associated within the current sliding window. For data blocks The normalized value of the cumulative cursor hover duration associated with the current sliding window. For data blocks Context matching degree between the current active build task and the current active build task. , , These are the editing frequency weight coefficient, the hover duration weight coefficient, and the construction context matching degree weight coefficient, respectively, and they satisfy the following conditions: , This is the index number of the data block.

[0036] The calculated intent intensity index for each data block is output as an intent intensity index sequence.

[0037] The above construction context matching degree The calculation method is as follows: obtain the build configuration file of the currently active CI / CD pipeline, parse the build configuration file to obtain the set of all dependency names declared in the build process, and determine the data block. Does it belong to a dependency in this set of dependency names? If it does, then... The value is 1; if it does not belong to the category, then... The value is 0. When multiple active pipelines exist simultaneously, the maximum value among the matching degree values ​​corresponding to each pipeline is taken as the [value]. .

[0038] The sliding window described above has a window length of 5 seconds and a sliding step size of 1 second. That is, every second, an aggregation calculation is performed on the events within the past 5 seconds, and the intent intensity index of each data block within the sliding window is output.

[0039] In order for the intent intensity index to reflect the temporal trend of access demand, the calculation of the intent intensity index... Furthermore, an exponentially weighted moving average was introduced. Specifically, the exponentially weighted moving average was calculated using the current sliding window. Smoothed intent intensity index compared to the output of the previous sliding window Perform weighted fusion to obtain the smoothed intent intensity index of the current sliding window. The calculation formula is: ; in, For data blocks The intention intensity index after smoothing in the current sliding window. This is the smoothing coefficient, and its value range is... , The intent intensity index is the original value calculated for the current sliding window. The superscript represents the smoothed intent intensity index of the previous sliding window output. and These represent the current sliding window time and the previous sliding window time, respectively, with the time interval between two adjacent times equal to the sliding step size. By using an exponentially weighted moving average, short-term fluctuations can be smoothed out while preserving the overall trend of access demand.

[0040] Step 4: Generate cross-layer hierarchical migration instructions based on the intent intensity index and the current residence level of the data block. For each data block in the intent intensity index sequence Query the global data routing table to obtain data blocks. Current residency level Combined with the intention intensity index Based on the preset hierarchical decision parameters, hierarchical migration instructions are generated according to the following decision process: Step 401, determine the intent intensity index Is it greater than the preset high intensity threshold? ,like Execute step 402; if Proceed to step 404.

[0041] For the range of values ​​within The dimensionless threshold within the normalized intention intensity index Consistent units are used to distinguish whether the current data block is in a state of high-frequency access demand.

[0042] Step 402, calculate the edge layer missing penalty value. The calculation formula is: ; in, For data blocks Edge layer missing penalty value, For data blocks The current intent intensity index of the sliding window is a dimensionless normalized value. This is the estimated transmission latency, in milliseconds, for downloading data blocks from the cloud layer to the edge layer. For data blocks The estimated number of visits within the current sliding window, in seconds. The unit of is milliseconds·times, which represents the weighted latency cost caused by the absence of data blocks at the edge layer.

[0043] The value is derived from: edge layer nodes periodically sending probe messages to the cloud layer, recording the round-trip time of the messages, and estimating the time required for data block transmission by combining historical transmission rates, and using the average value within the most recent statistical period as the current value. The value of . The value is derived from the data block within the current sliding window. The number of associated access intent events is estimated by combining the average number of actual accesses after similar events are triggered within the historical sliding window.

[0044] Step 403, adjust the edge layer missing penalty value. The edge storage cost of storing this data block at the edge layer Comparison. Due to the missing penalty value at the edge layer. Edge layer storage footprint cost Since their dimensions are different, they need to be normalized separately using mean normalization based on the range to scale them to the appropriate values. After comparing the intervals, normalize the edge layer missing penalty values ​​corresponding to all data blocks to be decided within the current decision period. Edge layer storage footprint cost Executed independently across the distribution. If normalized... and If it is the cloud layer, then a data sinking instruction is generated, the instruction content of which is to sink the data block. Migrate from the cloud layer to the edge layer node that initiated the access request; if normalized... and For the edge layer, the terminal layer simultaneously issues a request for the data block. If the deep working signal of the relevant code module is triggered, a terminal pre-push instruction will be generated, the instruction content of which is to push data blocks... The cache is then migrated from the edge layer to the local cache of the corresponding terminal device.

[0045] Step 404, Obtain data block Continuous stay at the current level of residence and global access frequency attenuation ratio The global access frequency attenuation ratio For data blocks The ratio of recent global access frequency to historical peak global access frequency is a dimensionless value, and its range is [value missing]. .like Exceeding the preset survival time threshold and Below the preset attenuation threshold Then a data eviction instruction is generated, the instruction content of which is to evict the data block. Migrate from the current edge layer nodes to the cloud layer cold storage area to free up high-speed storage space at the edge layer.

[0046] For the range of values ​​within The dimensionless threshold within, and The units are consistent, and it is used to determine whether the access frequency of a data block has decayed to the point that it is no longer worthwhile to continue occupying high-speed storage space in the edge layer; The duration threshold is in seconds, and Dimensional consistency is used to limit the maximum residence time of data blocks at the edge layer. Both conditions must be met simultaneously to trigger a data eviction command, in order to avoid eviction of data blocks that are still in a high-frequency access state simply because the residence time limit has been exceeded.

[0047] The calculation method is as follows: the cloud layer aggregates the data blocks from each edge layer node and terminal layer device within the current statistical period. The total number of access requests generated is used as a count value for the recent global access frequency; data blocks are also maintained. The historical maximum value of the total number of access requests within each statistical period since it was first included in the hierarchical management is used as the count value of the historical peak global access frequency; dividing the two yields the result. When the historical peak count is zero, The value is 0.

[0048] Step 405: Summarize the hierarchical migration instructions generated by each of the above decision branches to form a hierarchical migration instruction set for the current decision cycle. Each hierarchical migration instruction in the hierarchical migration instruction set includes a unique data block identifier, a source level identifier, a target level identifier, and a target node network address.

[0049] The method for determining the aforementioned depth working signal is as follows: when the frequency of editing operation events generated by the terminal layer for the data block set associated with the same code module in multiple consecutive sliding windows is continuously higher than the preset depth working frequency threshold, the terminal layer generates a depth working signal and reports it to the edge layer.

[0050] The aforementioned edge layer storage costs The calculation method is as follows: ,in For data blocks Storage space size, This refers to the remaining high-speed storage capacity currently available to the edge layer node; both are in the same unit. This is a dimensionless ratio. The smaller the remaining capacity of the edge layer, the higher the storage cost of the edge layer, thus raising the threshold for determining data sinking.

[0051] Building upon steps 401 to 405, the process also includes dependency-based data sinking. Specifically, after the data sinking instruction is generated in step 403, the data block is retrieved. A dependency graph is generated by parsing the dependency description file of the software project, and represents the dependencies between data blocks in a directed acyclic graph structure. The data blocks in the dependency graph are traversed. direct dependencies ,in For each direct dependency, the index number is used. Query direct dependencies The current residency level, if directly dependent on items Currently residing in the cloud layer, it will directly depend on the item. Add to the hierarchical migration instruction set for the current decision cycle to generate instructions for direct dependencies. The associated sinking instruction. Through dependency-related sinking processing, in data blocks... When sinking to the edge layer, the data block Directly dependent data blocks are also pushed down to avoid subsequent access to the data blocks. The cloud fetch was triggered again due to missing dependencies.

[0052] To improve the data preheating efficiency of the edge layer in pipeline build scenarios, in addition to step 4, the following steps are also included: When the edge layer CI / CD node receives a pipeline trigger event, it parses the dependency description file referenced in the pipeline configuration, generates a directed acyclic dependency graph, and queries the current residence level of each dependency in the directed acyclic dependency graph according to the global data routing table. For dependency data blocks residing in the cloud layer, without waiting for the build script to execute to the corresponding download step, a streaming preheating transmission instruction is directly generated to transmit these dependency data blocks to the edge layer cache in advance. For dependency data blocks already residing in the edge layer, when the edge layer cache manager intercepts the download request issued by the build tool, it compares the content hash value in the download request with the hash value of the locally cached data block. If they match, it directly returns the local storage path and does not initiate a download request to the cloud.

[0053] Step 5: Perform cross-layer migration of data blocks and atomically update the global data routing table. Read the hierarchical migration instruction set for the current decision cycle, and for each hierarchical migration instruction in the hierarchical migration instruction set, perform the data block migration operation according to the following sub-steps: Step 501: Create data blocks on the target nodes at the target level. A temporary copy is created, which is stored as a shadow file in the temporary storage area of ​​the target node. During the shadow file writing process, data blocks in the global data routing table are... The storage location mapping remains unchanged, pointing to the original storage location at the source level.

[0054] Step 502: Perform integrity verification on the shadow file on the target node by comparing the content hash value of the shadow file with the content hash value of the data block recorded in the global data routing table. If the hash values ​​match, the verification passes and proceed to step 503; if the hash values ​​do not match, delete the shadow file, mark the hierarchical migration instruction as failed, and record it in the failure log.

[0055] Step 503: Add data blocks from the global data routing table. The storage location mapping is modified to the network address of the target node, the shadow file is renamed to the official data file, the policy version number is incremented by 1, and the updated routing entries are broadcast to each edge layer node and terminal layer device.

[0056] Step 504: For data sinking hierarchical migration instructions, retain the data blocks at the source level. The original copy is used as a disaster recovery backup; for data eviction-type hierarchical migration commands, after the global data routing table is updated and it is confirmed that each level node has synchronized the new version of the global data routing table information, the data block copy on the source level is deleted to release storage space.

[0057] In step 503 above, the global data routing table modification and shadow file renaming operations are executed atomically, meaning that the two operations either complete simultaneously or neither is executed at all. If either operation fails, the system rolls back to the state before the operation, and the data blocks... The storage location mapping still points to the original storage location at the source level, ensuring that developers or pipeline builders will not obtain corrupted or incomplete data when reading data blocks during the migration process.

[0058] To perform online correction of the hierarchical decision parameters based on actual access deviations, the following streaming negative feedback correction step is included after step 5 is completed: At the end of each statistical period, the edge layer counts the number of two types of deviation events: the first type is the number of data blocks that were predicted to be accessed but were not actually accessed, denoted as the invalid warm-up number. The second category is the number of data blocks that were not predicted but were actually accessed frequently, denoted as cache penetration number. .

[0059] The statistical method is as follows: At the end of each statistical period, iterate through all data blocks that were moved down to the edge layer or pre-pushed to the terminal layer due to hierarchical migration decisions during the current statistical period, and count the total number of data blocks that were actually accessed zero times during that statistical period. The count value. The statistical method is as follows: At the end of each statistical period, all data blocks that were actually accessed during the current statistical period are traversed. The total number of data blocks that resided in the cloud layer at the beginning of the current statistical period and were not predicted as having high access demand by the tiered migration decision, but whose actual access frequency exceeds the preset access frequency threshold, is counted as this number. The count value.

[0060] The edge layer uses the FTRL online learning algorithm to calculate the edit frequency weighting coefficient in the intent intensity index. Hover duration weighting coefficient Constructing context matching degree weight coefficients Incremental updates are performed. The FTRL online learning algorithm uses... and As the input of the loss feedback signal, Corresponding to excessive preheating loss, Corresponding to the preheating loss, the two types of loss signals together constitute the loss function value of the current statistical period. The FTRL online learning algorithm calculates the editing frequency weight coefficient based on this loss function value. Hover duration weighting coefficient Constructing context matching degree weight coefficients The gradient, combined with the historical gradient accumulation, is used to weight the edit frequency coefficient. Hover duration weighting coefficient Constructing context matching degree weight coefficients Apply and After applying regularization constraints, the optimal update amount is calculated, and the output is the edit frequency weight coefficient. Hover duration weighting coefficient Constructing context matching degree weight coefficients The incremental update amount, the editing frequency weight coefficient is completed locally at the edge. Hover duration weighting coefficient Constructing context matching degree weight coefficients Incremental adjustments can be made without uploading the original access logs to the cloud layer.

[0061] The edge layer reports the anonymized deviation statistics to the cloud layer. These anonymized deviation statistics only include… and The values ​​and corresponding edge node identifiers are provided, but do not include specific data block identifiers or access log content. The cloud layer aggregates the deviation statistics reported by each edge node, when globally... When the cumulative value exceeds the preset upper limit for invalid preheating, the high-intensity threshold is increased. Tighten the conditions for data sinking; when the global scope When the cumulative value exceeds the preset penetration limit, the high-intensity threshold is reduced. The conditions for data migration to lower layers are relaxed. After the adjustment is completed, the strategy version number is incremented, and the updated hierarchical decision parameters are distributed to each edge layer node for use in the next round of hierarchical migration decisions.

[0062] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0063] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0064] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0065] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cloud computing-based software development data layering processing method, characterized in that, include: The configuration information of each node in the cloud layer, edge layer, and terminal layer is obtained, and a global data routing table is generated based on the configuration information. The global data routing table records the routing entry for each data block. Each routing entry includes a unique identifier for the data block, a hash value of the data block content, a current storage location mapping, and a policy version number. Access intent events related to software development operations are collected from the cloud layer, edge layer, and terminal layer respectively. The events collected from each layer are encapsulated in a unified event format to form a multi-source access intent event stream. The multi-source access intent event stream is aggregated using a sliding window. Within each sliding window, for each data block involved, feature values ​​of three dimensions—edit frequency, cumulative hover duration, and construction context matching degree—are extracted. After normalizing the feature values, they are weighted and summed to calculate the intent intensity index of each data block. For each data block, the global data routing table is queried to obtain the current resident level. The intent strength index is compared with a preset high-intensity threshold. When the intent strength index is greater than the high-intensity threshold, the edge layer missing penalty value and edge layer storage cost are calculated. The edge layer missing penalty value is equal to the product of the data block's intent strength index, the estimated transmission delay from the cloud layer to the edge layer, and the estimated number of accesses to the data block within the current sliding window. The estimated transmission delay is calculated by the edge layer node periodically sending probe messages to the cloud layer, recording the round-trip time of the messages, and estimating the time required for data block transmission by combining historical transmission rates, taking the average value within the most recent statistical period. The edge layer storage cost is equal to the ratio of the data block's storage space size to the remaining high-speed storage capacity available to the current edge layer node. The edge layer storage costs are compared after being normalized to the mean based on the range. If the normalized edge layer missing penalty value is greater than the normalized edge layer storage cost and the data block is currently residing at the cloud layer, a data sinking instruction is generated to migrate the data block from the cloud layer to the edge layer node that initiated the access request. If the normalized edge layer missing penalty value is greater than the normalized edge layer storage cost and the data block is currently residing at the edge layer, and the terminal layer issues a deep working signal for the code module to which the data block belongs, a terminal pre-push instruction is generated to migrate the data block from the edge layer to the local cache of the corresponding terminal layer device. The conditions for generating the deep working signal are: the frequency of editing operation events generated by the terminal layer for the set of data blocks associated with the same code module is continuously higher than a preset deep working frequency threshold within multiple consecutive sliding windows. When the intent intensity index is not greater than the high intensity threshold, the continuous residence time and global access frequency decay ratio of the data block at the current residence level are obtained. The global access frequency decay ratio is the ratio of the recent global access frequency of the data block to the historical peak global access frequency. The recent global access frequency is the total number of access requests generated by each edge layer node and terminal layer device for the data block in the current statistical period. The historical peak global access frequency is the historical maximum value of the total number of access requests in each statistical period since the data block was first included in the hierarchical management. When the historical peak count is zero, the global access frequency decay ratio is zero. When the continuous residence time exceeds the preset survival time threshold and the global access frequency decay ratio is lower than the preset decay threshold, a data eviction instruction is generated to migrate the data block from the current edge layer node to the cloud layer cold storage area, forming a hierarchical migration instruction set. The hierarchical migration instruction set is read, a shadow file of the data block is created on the target node, and the integrity of the content hash value is verified. After the verification is passed, the storage location mapping of the global data routing table is updated atomically, and the shadow file is converted into a formal data file.

2. The cloud computing-based software development data layering processing method according to claim 1, characterized in that, The global data routing table is persistently stored in the cloud layer, and an initial snapshot of the global data routing table is distributed to each edge layer node and terminal layer device. The policy version number is a monotonically increasing integer value. After each layer migration decision is completed and the global data routing table is updated, the cloud layer increments the policy version number by one, generates an incremental change of the global data routing table containing the new policy version number, and pushes it to each edge layer node and terminal layer device. After receiving an incremental change to the global data routing table, edge layer nodes and terminal layer devices compare the version number with the locally cached global data routing table. Only when the received policy version number is greater than the local version number will the incremental change be applied to the local copy of the global data routing table.

3. The cloud computing-based software development data layering processing method according to claim 1, characterized in that, Access intent events related to software development operations are collected from the cloud layer, edge layer, and terminal layer, including: At the terminal layer, the developer's operation events are captured through plug-in programs deployed in the integrated development environment. These operation events include cursor hovering over code symbol names, Git branch switching events, and unit test start events. Each operation event carries a session identifier, a timestamp, and the name of the operation object. At the edge layer, pipeline trigger events and dependency description file parsing events are captured by parsing the CI / CD pipeline configuration file. Each event carries a build identifier, code commit hash value, and the name of the dependency involved. At the cloud layer, event notifications from the code repository are received through the API gateway, capturing merge request creation events and version tag creation events. Each event carries the repository identifier, source branch name, and target branch name.

4. The cloud computing-based software development data layering processing method according to claim 3, characterized in that, Before encapsulating the events according to a unified event format, noise filtering is performed on the raw events collected by the terminal layer. The noise filtering process includes: for each terminal session, within a continuous time window, counting the triggering frequency of the same type of events on the same operation object. When the triggering frequency exceeds a preset upper limit threshold and the operation object between events has not changed, the repeated events within the time window are merged, and only the last event is retained as a valid event and output to the subsequent processing flow.

5. The cloud computing-based software development data layering processing method according to claim 1, characterized in that, The calculation of the intent intensity index for each data block includes: Within each sliding window, the editing frequency and cumulative hover duration are respectively processed by mean normalization based on the range, scaling them to the range of [0.1]. The context matching score is determined by whether the data block belongs to the set of dependency names declared in the current active CI / CD pipeline build configuration file. If it belongs, the value is one; otherwise, the value is zero. When multiple active pipelines exist at the same time, the maximum value of the matching score values ​​of each pipeline is taken. The intent intensity index is equal to the sum of the product of the edit frequency weight coefficient and the normalized edit frequency value, the product of the hover duration weight coefficient and the normalized hover cumulative duration value, and the product of the construction context matching degree weight coefficient and the construction context matching degree value, wherein the sum of the edit frequency weight coefficient, the hover duration weight coefficient, and the construction context matching degree weight coefficient is equal to one. The intention intensity index calculated by the current sliding window is subjected to an exponentially weighted moving average with the smoothed intention intensity index output by the previous sliding window to obtain the smoothed intention intensity index of the current sliding window. The smoothed intention intensity index is equal to the product of the smoothing coefficient and the original intention intensity index of the current sliding window plus a subtraction of the product of the smoothing coefficient and the smoothed intention intensity index of the previous sliding window.

6. The cloud computing-based software development data layering processing method according to claim 1, characterized in that, After the data sinking instruction is generated, the process also includes dependency association sinking: obtaining the dependency association graph of the data block, which is generated by parsing the dependency description file of the software project and represents the dependency relationship between each data block in a directed acyclic graph structure; traversing the direct dependencies of the data blocks in the dependency association graph, and for the direct dependencies currently residing in the cloud layer, generating a related sinking instruction for the direct dependency and appending it to the hierarchical migration instruction set of the current decision cycle.

7. The cloud computing-based software development data layering processing method according to claim 1, characterized in that, After performing cross-layer migration of data blocks and updating the global data routing table, a streaming negative feedback correction step is also included: At the end of each statistical period, the edge layer counts the number of invalid preheating and cache penetration. The number of invalid preheating is the total number of data blocks that are sinked to the edge layer or pre-pushed to the terminal layer due to the hierarchical migration decision in the current statistical period but have zero actual access counts. The number of cache penetration is the total number of data blocks that reside in the cloud layer in the current statistical period and are not predicted to be in high access demand but have more than a preset access count threshold. The edge layer employs the FTRL online learning algorithm, using the number of invalid warm-ups and the number of cache penetrations as input loss feedback signals to incrementally update the editing frequency weight coefficient, hover duration weight coefficient, and construction context matching degree weight coefficient in the intent intensity index calculation.

8. A cloud computing-based software development data layering processing system, comprising the cloud computing-based software development data layering processing method according to any one of claims 1-7, characterized in that, include: The global routing initialization module is used to obtain the configuration information of each node in the cloud layer, edge layer and terminal layer, and generate a global data routing table based on the configuration information. The global data routing table records the routing entry for each data block. Each routing entry includes a unique identifier for the data block, a hash value of the data block content, a current storage location mapping and a policy version number. The multi-source event acquisition module is used to collect access intent events related to software development operations from the cloud layer, edge layer, and terminal layer respectively, and encapsulate the events collected at each layer according to a unified event format to form a multi-source access intent event stream. The intent intensity calculation module is used to perform sliding window aggregation processing on the multi-source access intent event stream. Within each sliding window, for each data block involved, it extracts feature values ​​of three dimensions: editing frequency, cumulative hover duration, and construction context matching degree. After normalizing the feature values, it performs weighted summation to calculate the intent intensity index of each data block. The migration decision generation module is used to query the global data routing table for each data block to obtain the current resident level, compare the intent strength index with a preset high-intensity threshold, and calculate the edge layer missing penalty value and edge layer storage cost when the intent strength index is greater than the high-intensity threshold. The edge layer missing penalty value is equal to the product of the data block's intent strength index, the estimated transmission delay from the cloud layer to the edge layer, and the estimated number of accesses to the data block within the current sliding window. The estimated transmission delay is estimated by the edge layer node periodically sending probe messages to the cloud layer, recording the round-trip time of the messages, and combining the data with historical transmission rates. The time required for block transmission is taken as the average of the most recent statistical period. The edge layer storage cost is equal to the ratio of the data block's storage space size to the remaining high-speed storage capacity available to the current edge layer node. The edge layer missing penalty value and the edge layer storage cost are compared after being normalized to the mean based on the range. If the normalized edge layer missing penalty value is greater than the normalized edge layer storage cost and the data block's current resident level is the cloud layer, a data sinking instruction is generated to migrate the data block from the cloud layer to the edge layer node that initiated the access request. If the current residence level of the data block is the edge layer, and the terminal layer issues a deep working signal for the code module to which the data block belongs, a terminal pre-push instruction is generated to migrate the data block from the edge layer to the local cache of the corresponding terminal layer device. The deep working signal is generated under the following conditions: within multiple consecutive sliding windows, the frequency of editing operation events generated by the terminal layer for the set of data blocks associated with the same code module is continuously higher than a preset deep working frequency threshold; when the intent intensity index is not greater than the high intensity threshold, the continuous residence time of the data block at the current residence level and the global access frequency attenuation ratio are obtained, where the global access frequency attenuation ratio is the ratio of the recent global access frequency of the data block to the global access frequency of the data block. The ratio of historical peak global access frequency, where the recent global access frequency is the total number of access requests generated by each edge layer node and terminal layer device for the data block in the current statistical period, and the historical peak global access frequency is the historical maximum value of the total number of access requests in each statistical period since the data block was first included in hierarchical management. When the historical peak count is zero, the global access frequency decay ratio is zero. When the continuous residence time exceeds the preset lifetime threshold and the global access frequency decay ratio is lower than the preset decay threshold, a data eviction instruction is generated to migrate the data block from the current edge layer node to the cloud layer cold storage area, forming a hierarchical migration instruction set. The migration execution module is used to read the hierarchical migration instruction set, create a shadow file of the data block on the target node and perform content hash value integrity verification. After the verification is passed, it atomically updates the storage location mapping of the global data routing table and converts the shadow file into a formal data file.

Images