Software development scheme generation method and system

Abstract

The invention discloses a method and system for generating a software development scheme, and belongs to the technical field of software development, and the method comprises the steps: collecting system resource occupation data and thread operation data during multi-user concurrent operation, segmenting the system resource occupation data into resource fragments according to time dimensions, positioning resource competition points, and generating conflict mark data; constructing a thread execution graph based on the conflict mark data, identifying mutually exclusive resource points, constructing a double-layer distributed queue, setting a resource state monitor to dynamically allocate thread priorities, and generating scheduling processing codes; monitoring the running state of the scheduling processing code, and splitting the thread into a plurality of sub-threads for parallel execution when detecting that the thread is blocked; and collecting an execution result, recording the execution path with the shortest response time and the highest resource utilization rate as a scheduling template, and generating a software development scheme. According to the method, the utilization efficiency of system resources is improved through dynamic scheduling and parallel processing, and the thread blocking probability is reduced.

Description

Technical Field

[0001] This invention belongs to the field of software development technology, and in particular relates to a method and system for generating software development solutions. Background Technology

[0002] As software systems continue to grow in scale, resource contention issues in scenarios with multiple concurrent users become increasingly prominent. Existing technologies typically employ synchronization mechanisms such as mutexes or semaphores to resolve resource contention, but these methods can easily lead to frequent thread blocking and degrade system performance.

[0003] Current software development solutions suffer from insufficient precision in resource contention detection, failing to effectively identify system bottlenecks. Fixed thread scheduling strategies struggle to adapt to dynamic load changes, and simplistic thread blocking handling mechanisms easily lead to wasted computational resources. While priority-based dynamic scheduling methods have been proposed, the lack of real-time monitoring and analysis of system operation makes accurate prediction of resource contention difficult, resulting in suboptimal scheduling performance. Existing solutions often overlook the performance impact of thread blocking, failing to fully utilize idle system resources. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for generating software development solutions, thus solving the aforementioned problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for generating a software development solution, comprising the following steps: Collect system resource usage data and thread execution data during multi-user concurrent operations, divide the system resource usage data into resource segments according to the time dimension, locate resource contention points in the resource segments, and associate the resource contention points with the thread execution data to generate conflict marker data; A thread execution graph is constructed based on conflict marker data. Mutual exclusive resource points are identified in the thread execution graph. A two-layer distributed queue is constructed based on the mutual exclusive resource points. Resource status listeners are set in the two-layer distributed queue. Thread priorities are dynamically allocated based on the resource status listeners, and scheduling processing code is generated. Monitor the running status of the scheduling code, record the thread blocking time, and when the thread blocking time exceeds the preset blocking time, split the blocked thread into multiple sub-threads, allocate the sub-threads to idle computing nodes for parallel execution, and generate parallel processing code. The execution results of parallel processing code are collected, the response time and resource utilization of sub-threads are calculated, the execution path with the shortest response time and the highest resource utilization is recorded as a scheduling template, the system resource allocation strategy is determined based on the scheduling template, and a software development plan is generated.

[0006] Furthermore, system resource usage data and thread execution data are collected during multi-user concurrent operations. The system resource usage data is segmented into resource fragments along the time dimension. Resource contention points are located within these fragments, and these resource contention points are correlated with thread execution data to generate conflict marker data, including: Collect system resource usage data and thread execution data, calculate resource utilization rate from the system resource usage data, generate resource utilization curves according to the collection time sequence, and divide resource segments based on the fluctuation trend of the resource utilization curves; Identify abrupt changes in the resource usage curve within a resource segment, mark these abrupt changes as resource contention points, extract thread execution data at the time of the resource contention point into thread combinations, calculate the access interval of each thread in the thread combination to the same resource, and generate access time sequence data. The overlapping intervals of thread access in the access time sequence data are marked, the resource occupation time of the threads in the overlapping interval is calculated, and when the resource occupation time is longer than the preset occupation time, the corresponding thread group is marked as a competition group and conflict marking data is generated.

[0007] Furthermore, a thread execution graph is constructed based on the conflict marker data. Mutual exclusive resource points are identified in the thread execution graph. A two-level distributed queue is constructed based on the mutual exclusive resource points. Resource status listeners are set in the two-level distributed queue. Thread priorities are dynamically allocated based on the resource status listeners, and scheduling processing code is generated, including: A thread execution graph is constructed based on conflict marker data. Threads in the thread execution graph are set as nodes, and resource access relationships between threads are set as connecting edges. The resource contention value of the connecting edges is calculated, and thread association data is generated. The resource access frequency of nodes is calculated based on thread-related data. Nodes whose resource access frequency exceeds a preset frequency value are marked as mutual exclusion resource points. A two-layer distributed queue is constructed based on the mutual exclusion resource points. A local data caching layer and a global data sharing layer are set in the two-layer distributed queue to generate queue structure data. Based on the queue structure data, resource status listeners are set up in the local data cache layer and the global data sharing layer respectively to monitor the length of the resource queue. When the queue length exceeds the preset length value, the resource access request is allocated to the local data cache layer. When the resource processing time exceeds the preset time value, the data is updated to the global data sharing layer to generate resource status data. Based on the resource status data collected in real time by the resource status listener, the ratio of resource occupancy time to preset base time is calculated periodically, the thread priority is dynamically updated according to the ratio, and scheduling processing code is generated based on the updated thread priority.

[0008] Furthermore, a two-layer distributed queue is constructed based on mutually exclusive resource points. Within this two-layer distributed queue, a local data caching layer and a global data sharing layer are set up, generating queue structure data including: The spatial distribution and access relationships between mutually exclusive resource points are statistically analyzed. Mutually exclusive resource points whose spatial distance is less than a first threshold and whose access frequency is greater than a second threshold are grouped into resource blocks. The data capacity and access popularity of the resource blocks are calculated to generate resource distribution data. A two-layer distributed queue is constructed based on resource distribution data. A local data cache layer and a global data sharing layer are set in the two-layer distributed queue. The local data cache layer is divided into a write buffer and a read buffer. Data fragmentation and transmission channels are established in the global data sharing layer. Set the status flag of the resource block in the write buffer and read buffer, configure the index table of the resource block in the data fragment, map the status flag and index table to the data access channel, and generate the data access policy. According to the data access strategy, the local data cache layer is configured as a fast access queue, the global data sharing layer is configured as a persistent queue, the resource blocks are allocated to the corresponding queue levels, and queue structure data is generated.

[0009] Furthermore, the execution status of the scheduling code is monitored, and the thread blocking duration is recorded. When the thread blocking duration exceeds the preset blocking time, the blocked thread is split into multiple sub-threads, which are then allocated to idle computing nodes for parallel execution. The generated parallel processing code includes: Monitor the running status of the scheduling code, collect the number of thread switching and resource usage time, calculate the thread blocking duration, mark threads whose blocking duration exceeds the preset blocking time as blocked threads, and generate thread status data; Analyze the memory access sequence of blocked threads in the thread state data, divide the memory access sequence into continuous regions and discrete regions according to address continuity, construct computing tasks based on the continuous regions, construct distribution tasks based on the discrete regions, and generate a thread splitting scheme. Collect the processing performance and memory status of computing nodes, mark computing nodes with processing performance higher than a preset performance threshold and memory status greater than a preset memory threshold as idle computing nodes, and allocate sub-thread tasks to idle computing nodes based on the thread splitting scheme to generate a node allocation scheme. Based on the node allocation scheme, sub-thread tasks are deployed on idle computing nodes. Communication channels are established between sub-thread tasks to coordinate the parallel execution of sub-threads and generate parallel processing code.

[0010] Furthermore, the memory access sequences of blocked threads in the thread state data are analyzed. These sequences are divided into continuous and discrete regions based on address continuity. Computational tasks are constructed based on the continuous regions, and distribution tasks are constructed based on the discrete regions. The resulting thread splitting scheme includes: Analyze the memory access operations of blocked threads in the thread state data, obtain the execution order of the memory access operations, calculate the correlation between memory access operations based on the execution order, and generate a memory access sequence; The correlation degree of memory access operations in the memory access sequence is analyzed. Memory access operations with a correlation degree higher than a preset correlation threshold are divided into continuous regions, and memory access operations with a correlation degree lower than the preset correlation threshold are divided into discrete regions, generating region division information. Extract consecutive memory access operations from contiguous regions, organize the pipeline structure based on the execution order of consecutive memory access operations, build data transmission channels in the pipeline structure, and generate computation tasks. Extract discrete memory access operations from discrete regions, organize the cache structure based on the execution order of discrete memory access operations, build data exchange channels in the cache structure, and generate distribution tasks. The data transmission channel of the computing task is connected to the data exchange channel of the distribution task to form a data path, and a thread splitting scheme is generated based on the data path.

[0011] Furthermore, the execution results of parallel processing code are collected, the response time and resource utilization of sub-threads are calculated, and the execution path with the shortest response time and highest resource utilization is recorded as a scheduling template. Based on the scheduling template, the system resource allocation strategy is determined, and a software development plan is generated, including: Collect the execution results of parallel processing code, record the start and end times of sub-threads, monitor the memory usage and processor utilization of sub-threads, calculate the runtime based on the time difference, calculate the resource consumption value based on usage and utilization, and combine the runtime and resource consumption value to generate performance index data. The runtime in the performance metrics data is marked as response time, and the resource consumption value is marked as resource utilization rate. A two-dimensional performance evaluation matrix is ​​constructed based on the response time and resource utilization rate. The execution paths in the two-dimensional performance evaluation matrix are sorted to generate path analysis data. Extract the execution path with the shortest response time and highest resource utilization from the path analysis data, record the thread allocation method and resource configuration parameters of the execution path, record the execution path as a scheduling template, and generate scheduling template data; The scheduling template data is converted into a system resource allocation strategy. Based on the system resource allocation strategy, thread priorities and resource quotas are set to generate a software development plan.

[0012] This invention provides a software development solution generation system, the system comprising: The data acquisition module is used to collect system resource usage data and thread execution data during multi-user concurrent operations. It divides the system resource usage data into resource segments according to the time dimension, locates resource contention points in the resource segments, and associates the resource contention points with the thread execution data to generate conflict marker data. The scheduling processing module is used to construct a thread execution graph based on conflict marker data, identify mutual exclusion resource points in the thread execution graph, construct a two-level distributed queue based on the mutual exclusion resource points, set resource status listeners in the two-level distributed queue, dynamically allocate thread priorities based on the resource status listeners, and generate scheduling processing code. The parallel processing module is used to monitor the running status of the scheduling processing code, record the thread blocking time, and when the thread blocking time is detected to exceed the preset blocking time, the blocked thread is split into multiple sub-threads, and the sub-threads are allocated to idle computing nodes for parallel execution to generate parallel processing code. The scheme generation module is used to collect the execution results of parallel processing code, calculate the response time and resource utilization of sub-threads, record the execution path with the shortest response time and the highest resource utilization as a scheduling template, determine the system resource allocation strategy based on the scheduling template, and generate a software development scheme.

[0013] This invention improves the accuracy of resource contention detection by collecting system resource usage data and thread execution data, accurately locating resource contention points, and establishing conflict marker data. Based on a thread execution graph and a two-layer distributed queue design, it achieves fine-grained management of system resources, reducing resource contention between threads. Dynamically adjusting thread priorities through a resource status listener enables the system to respond quickly to load changes, improving resource utilization efficiency. When thread blocking is detected, it is split into multiple sub-threads and allocated to idle computing nodes for execution, avoiding waste of computing resources. Attached Figure Description

[0014] Figure 1 A flowchart illustrating a method for generating a software development scheme according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a software development scheme generation system provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] like Figure 1 As shown, Figure 1 A flowchart illustrating a method for generating a software development solution according to an embodiment of the present invention is provided. The method includes the following steps: Collect system resource usage data and thread execution data during multi-user concurrent operations, divide the system resource usage data into resource segments according to the time dimension, locate resource contention points in the resource segments, and associate the resource contention points with the thread execution data to generate conflict marker data; A thread execution graph is constructed based on conflict marker data. Mutual exclusive resource points are identified in the thread execution graph. A two-layer distributed queue is constructed based on the mutual exclusive resource points. Resource status listeners are set in the two-layer distributed queue. Thread priorities are dynamically allocated based on the resource status listeners, and scheduling processing code is generated. Monitor the running status of the scheduling code, record the thread blocking time, and when the thread blocking time exceeds the preset blocking time, split the blocked thread into multiple sub-threads, allocate the sub-threads to idle computing nodes for parallel execution, and generate parallel processing code. The execution results of parallel processing code are collected, the response time and resource utilization of sub-threads are calculated, the execution path with the shortest response time and the highest resource utilization is recorded as a scheduling template, the system resource allocation strategy is determined based on the scheduling template, and a software development plan is generated.

[0017] Collect system resource usage data and thread execution data during multi-user concurrent operations. Divide the system resource usage data into resource segments based on time. Locate resource contention points within these segments and correlate these points with thread execution data to generate conflict marker data, including: Collect system resource usage data and thread execution data, calculate resource utilization rate from the system resource usage data, generate resource utilization curves according to the collection time sequence, and divide resource segments based on the fluctuation trend of the resource utilization curves; Identify abrupt changes in the resource usage curve within a resource segment, mark these abrupt changes as resource contention points, extract thread execution data at the time of the resource contention point into thread combinations, calculate the access interval of each thread in the thread combination to the same resource, and generate access time sequence data. The overlapping intervals of thread access in the access time sequence data are marked, the resource occupation time of the threads in the overlapping interval is calculated, and when the resource occupation time is longer than the preset occupation time, the corresponding thread group is marked as a competition group and conflict marking data is generated.

[0018] First, system resource usage data and thread execution data are collected, including resource metrics such as CPU utilization, memory usage, disk I / O, and network I / O, as well as thread execution information such as thread ID, thread state, thread priority, and thread locks. The data is collected 10 times per second to ensure the data's temporal accuracy meets analytical requirements.

[0019] Resource utilization rate is calculated from the collected system resource usage data. The resource utilization rate is calculated as the ratio of resource usage at a specific point in time to the total system resources. For example, if the total number of CPU cores is 8, and at a certain moment the CPU is using 6.4 cores, then the CPU utilization rate is 80%. The calculated resource utilization rates are arranged in chronological order of data collection to generate a resource utilization curve. The resource utilization curve is plotted with time on the horizontal axis and resource utilization rate on the vertical axis, forming a time-series graph of resource usage changes.

[0020] Resource segments are divided based on the fluctuation trend of resource usage curves, with the segmentation determined by changes in the curve's slope. A segment breakpoint is marked when the curve's slope changes beyond a preset threshold. The preset threshold is 10% / s, meaning a significant fluctuation is considered to occur when the resource usage rate changes by more than 10 percentage points per second. In this way, the entire monitoring period is divided into multiple resource segments, each with a relatively stable or consistent resource usage trend.

[0021] Within each segmented resource area, identify abrupt changes in the resource usage curve. The criterion for identifying an abrupt change is when the difference in resource usage between two adjacent time points exceeds a preset threshold. This threshold is set to 5%, meaning that a point is marked as an abrupt change when the difference in resource usage between two adjacent data points exceeds 5%. These identified abrupt changes are then marked as resource contention points; these points represent moments of drastic change in system resource usage, potentially indicating thread contention issues.

[0022] For each resource contention point, the thread execution data at that moment is extracted to form a thread group. The thread group includes information on all currently active threads, and each thread's information includes its thread ID, thread state, and the resource lock it holds. For example, at a certain resource contention point, there are three active threads in the system: threads A, B, and C. They are in the running, blocked, and waiting states, respectively. Thread A holds resource lock R1, thread B is waiting for resource lock R1, and thread C holds resource lock R2.

[0023] Calculate the access intervals of each thread in the computing thread combination to the same resource, and generate access timing data. Extract the resource access timestamps from the thread operation logs, and calculate the time difference between different threads accessing the same resource. For the case where thread A and thread B access resource R, if thread A accesses resource R at time point T1 and thread B accesses resource R at time point T2, the access interval is |T2 - T1|. Arrange the access operations of each thread to the resource in chronological order according to the calculated access intervals to form access timing data.

[0024] Mark the overlapping intervals of thread accesses that appear in the access timing data. The overlapping interval refers to the time period during which multiple threads overlap in accessing the same resource. Check the access timing data. If there is an intersection between the resource access time periods of two threads, mark this intersection as an overlapping interval. Thread A accesses resource R in the time period [T1, T3], and thread B accesses resource R in the time period [T2, T4], where T1 < T2 < T3 < T4, then the overlapping interval is [T2, T3].

[0025] Calculate the resource occupation duration of the threads within the overlapping interval. The resource occupation duration refers to the time that a thread holds the resource within the overlapping interval. Subtract the start time from the end time of the overlapping interval, and determine the actual resource occupation duration of each thread within the overlapping interval according to the resource acquisition and release records of the thread. In the above example, the resource occupation duration of thread A within the overlapping interval [T2, T3] is T3 - T2.

[0026] When the resource occupation duration is greater than the preset occupation time, mark the corresponding thread group as a competition group. The preset occupation time is set to 100 ms, that is, when a thread occupies the resource for more than 100 ms within the overlapping interval, it is determined as resource competition. Generate conflict marking data for the thread group marked as a competition group, including information such as the name of the competing resource, the list of competing thread IDs, the competing time period, and the degree of competition. The degree of competition is calculated according to the ratio of the resource occupation duration to the length of the overlapping interval. The higher this ratio, the more serious the degree of competition.

[0027] For example, after collecting system data, it was found that the CPU utilization rate rose sharply from 45% to 85% between 10:15:30 and 10:15:35, showing a clear fluctuation trend. This period was divided into a resource segment. Within this segment, the CPU utilization rate suddenly increased from 60% to 75% at 10:15:32, which was marked as a resource contention point. Extracting the thread execution data at this moment revealed that threads 1, 2, and 3 were simultaneously active and accessing database connection pool resources. Access timing analysis showed that threads 1 and 2 overlapped in accessing database connection resources within the time period [10:15:32.100, 10:15:32.350], with an overlap interval of 250ms. Thread 1 occupied connection resources for 180ms within the overlap interval, exceeding the preset occupation time threshold of 100ms. Therefore, threads 1 and 2 were marked as a contention group, generating corresponding conflict marker data.

[0028] This invention can automatically detect resource contention points during system operation, extract key thread combinations, analyze resource access conflicts between threads, and generate detailed thread contention data. It not only improves the efficiency and accuracy of thread contention detection but also reduces the workload of manual troubleshooting, optimizes system architecture, enhances software concurrency processing capabilities and response speed, and effectively solves the technical challenge of accurately locating thread contention problems using traditional performance analysis methods.

[0029] A thread execution graph is constructed based on conflict marker data. Mutual exclusive resource points are identified within the graph. A two-layer distributed queue is built based on these points. Resource status listeners are set within the queues. Thread priorities are dynamically allocated based on these listeners. Scheduling code is then generated, including: A thread execution graph is constructed based on conflict marker data. Threads in the thread execution graph are set as nodes, and resource access relationships between threads are set as connecting edges. The resource contention value of the connecting edges is calculated, and thread association data is generated. The resource access frequency of nodes is calculated based on thread-related data. Nodes whose resource access frequency exceeds a preset frequency value are marked as mutual exclusion resource points. A two-layer distributed queue is constructed based on the mutual exclusion resource points. A local data caching layer and a global data sharing layer are set in the two-layer distributed queue to generate queue structure data. Based on the queue structure data, resource status listeners are set up in the local data cache layer and the global data sharing layer respectively to monitor the length of the resource queue. When the queue length exceeds the preset length value, the resource access request is allocated to the local data cache layer. When the resource processing time exceeds the preset time value, the data is updated to the global data sharing layer to generate resource status data. Based on the resource status data collected in real time by the resource status listener, the ratio of resource occupancy time to preset base time is calculated periodically, the thread priority is dynamically updated according to the ratio, and scheduling processing code is generated based on the updated thread priority.

[0030] First, a thread execution graph is constructed using conflict marker data for analysis and optimization. Based on the conflict marker data obtained in the previous step, a thread execution graph is constructed, with threads set as nodes in the graph and resource access relationships between threads set as connecting edges. During the construction of the thread execution graph, each thread is represented as a node in the graph, and the node contains attributes such as thread ID, thread name, and thread state; the resource access relationships between threads are represented as connecting edges between nodes, and the connecting edges contain attributes such as resource type, access method, and access time.

[0031] The resource contention value of a connection edge is calculated based on factors such as the overlap time of threads accessing the same resource, the time the resource is locked, and the frequency of resource requests. Specifically, for a connection edge between two threads A and B, if their overlap time accessing resource R is T_overlap, the total time resource R is locked is T_locked, and the frequency of resource R requests is F_requests, then the resource contention value is T_overlap multiplied by T_locked multiplied by F_requests. After the resource contention value is calculated, thread association data is generated, including node information, connection edge information, and the resource contention value.

[0032] Resource access frequency is calculated based on thread-related data. The calculation method for resource access frequency is to count the number of times a thread accesses a specific resource per unit of time. For node A, if it accesses resource R 50 times within 10 seconds, the resource access frequency is 5 times / second. Nodes with resource access frequencies exceeding a preset frequency value are marked as mutually exclusive resource points. The preset frequency value is set to 3 times / second, meaning that when a node's resource access frequency exceeds 3 times / second, that node is marked as a mutually exclusive resource point.

[0033] A two-layer distributed queue is constructed based on mutually exclusive resource points. The two-layer distributed queue consists of a local data caching layer and a global data sharing layer. The local data caching layer resides on the local server where the application resides and is used to cache frequently accessed resource data; the global data sharing layer resides in the shared storage area of ​​the distributed system and is used to store and synchronize global resource data. The construction process of the two-layer distributed queue involves creating a local cache queue and a global shared queue, setting parameters such as the queue's data structure, capacity limits, and expiration policies, and generating queue structure data. The queue structure data includes information such as queue type, queue capacity, data access interface, and synchronization strategy.

[0034] Based on the queue structure, resource status listeners are set up in both the local data caching layer and the global data sharing layer. The resource status listener uses a dedicated monitoring thread to periodically check the queue status, including metrics such as queue length and resource processing time. The listener monitors the resource queue length; when the queue length exceeds a preset value, resource access requests are allocated to the local data caching layer. The preset length value is set to 100, meaning that when the number of requests waiting to be processed in the global shared queue exceeds 100, new resource access requests will be preferentially allocated to the local cache queue, reducing the load on the global shared queue.

[0035] When resource processing time exceeds a preset time value, the data is updated to the global data sharing layer. The preset time value is set to 200ms, meaning that when the resource data processing time in the local cache exceeds 200ms, the processing result is updated to the global sharing layer, ensuring data consistency and visibility. The resource status listener collects resource status data in real time, including metrics such as queue length, resource processing time, and resource access frequency, and generates resource status data.

[0036] Based on real-time resource status data collected by a resource status listener, the ratio of resource usage time to a preset baseline time is calculated periodically. The periodic calculation frequency is once per second, and the preset baseline time is set to 100ms. The ratio of actual resource usage time to the preset baseline time is calculated by dividing the actual resource usage time by the preset baseline time. If a thread's actual resource usage time is 150ms and the preset baseline time is 100ms, the ratio is 1.5. Thread priorities are dynamically updated based on the calculated ratio. The update rule is: the larger the ratio, the lower the thread priority; the smaller the ratio, the higher the thread priority. When the ratio is less than 0.8, the thread priority increases by one level; when the ratio is greater than 1.2, the thread priority decreases by one level; when the ratio is between 0.8 and 1.2, the thread priority remains unchanged.

[0037] Scheduling code is generated based on the updated thread priorities. The process involves creating thread scheduling strategy code snippets based on the updated thread priorities, including setting thread priorities, adjusting thread scheduling order, and adjusting resource allocation strategies. The scheduling code utilizes thread control interfaces specific to the programming language, such as function calls for setting thread priorities and configuring thread scheduling strategies.

[0038] For example, a thread execution graph is constructed using conflict-marked data. It is found that threads A, B, and C compete for database connection resources, with resource contention values ​​for the connection edges: AB: 8.5, BC: 6.2, and AC: 4.8, respectively. The resource access frequency of the nodes is calculated: thread A: 4.2 times / s, thread B: 3.8 times / s, and thread C: 2.5 times / s. Since the resource access frequency of threads A and B exceeds the preset frequency value of 3 times / s, they are marked as mutually exclusive resource points. A two-layer distributed queue is constructed based on these mutually exclusive resource points, with the local data cache layer capacity set to 200 and the global data sharing layer capacity set to 500. After setting a resource status listener, if the global shared queue length reaches 120, exceeding the preset length value of 100, the system automatically allocates new resource access requests to the local cache layer for processing. If a resource processing time reaches 250ms, exceeding the preset time value of 200ms, the data is automatically updated to the global shared layer. The ratio of thread A's resource usage time to the preset baseline time is calculated periodically. If the ratio is 1.8, which exceeds 1.2, thread A's priority is reduced from 10 to 9. If the ratio for thread B is 0.7, which is less than 0.8, thread B's priority is increased from 8 to 9. Based on the updated thread priorities, scheduling code is generated, achieving dynamic optimization of resource contention.

[0039] This embodiment can accurately identify thread contention hotspots in the system, automatically build a highly adaptable resource access queue, and achieve intelligent distribution and load balancing of resource requests, significantly improving the system's concurrent processing capabilities and response speed. By monitoring resource status in real time and dynamically adjusting thread priorities, it can adaptively optimize resource allocation strategies based on system load conditions, reducing the risk of thread blocking and deadlock, and improving resource utilization efficiency.

[0040] A two-layer distributed queue is constructed based on mutually exclusive resource points. A local data caching layer and a global data sharing layer are set up within the two-layer distributed queue. The generated queue structure data includes: The spatial distribution and access relationships between mutually exclusive resource points are statistically analyzed. Mutually exclusive resource points whose spatial distance is less than a first threshold and whose access frequency is greater than a second threshold are grouped into resource blocks. The data capacity and access popularity of the resource blocks are calculated to generate resource distribution data. A two-layer distributed queue is constructed based on resource distribution data. A local data cache layer and a global data sharing layer are set in the two-layer distributed queue. The local data cache layer is divided into a write buffer and a read buffer. Data fragmentation and transmission channels are established in the global data sharing layer. Set the status flag of the resource block in the write buffer and read buffer, configure the index table of the resource block in the data fragment, map the status flag and index table to the data access channel, and generate the data access policy. According to the data access strategy, the local data cache layer is configured as a fast access queue, the global data sharing layer is configured as a persistent queue, the resource blocks are allocated to the corresponding queue levels, and queue structure data is generated.

[0041] First, the spatial distribution and access relationships between mutex resources are statistically analyzed. Spatial distribution refers to the relative positional relationship of mutex resources in the memory address space, obtained by calculating the memory address difference between resource points. Access relationship refers to the access dependency and order between different mutex resources, obtained by analyzing the call chain and timing relationship of resource access. Mutex resources with a spatial distance less than a first threshold and an access frequency greater than a second threshold are grouped into resource blocks. Optionally, the first threshold is set to 4KB, indicating that mutex resources with a memory address difference of less than 4KB are considered to be spatially close; the second threshold is set to 2 times / s, indicating that mutex resources with a mutual access frequency exceeding 2 times per second have a close access relationship.

[0042] Calculate the data capacity and access frequency of a resource block. Data capacity is calculated as the total memory occupied by all mutually exclusive resource points within the block. Access frequency is calculated as the total number of times all threads access the resource block per unit time. For example, if a resource block contains mutually exclusive resource points R1, R2, and R3, with memory occupancy of 1KB, 2KB, and 1.5KB respectively, then the data capacity of this resource block is 4.5KB. Within a 10-second time window, the total number of accesses to this resource block by all threads is 85, therefore the access frequency is 8.5 times / second. Generate resource distribution data, including the resource block identifier, a list of contained mutually exclusive resource points, spatial distribution information, an access relationship graph, data capacity, and access frequency.

[0043] A two-tiered distributed queue is constructed based on resource distribution data, comprising a local data caching layer and a global data sharing layer. The local data caching layer is deployed locally on the application server using in-memory storage, offering high-speed read / write capabilities and is suitable for caching frequently accessed resource data. The global data sharing layer is deployed in a distributed storage system using persistent storage, ensuring data consistency and suitable for storing resource data that needs to be shared across nodes. The local data caching layer is divided into a write buffer and a read buffer. The write buffer temporarily stores resource data to be written, employing a write merging strategy to reduce the number of write operations. The read buffer temporarily stores resource data retrieved from the global layer, employing a read-ahead strategy to improve data access speed. In the global data sharing layer, data sharding and a transmission channel are established. Data sharding divides large resource blocks into multiple smaller blocks for distributed storage, improving parallel processing capabilities. The transmission channel connects the local caching layer and the global sharing layer, enabling data synchronization and consistency maintenance.

[0044] Status flags for resource blocks are set in the write and read buffers. These flags include dirty flags, expired flags, and locked flags. The dirty flag indicates whether resource data has been modified but not yet written back to the global layer; the expired flag indicates whether cached data has expired and needs updating; and the locked flag indicates whether a resource is exclusively accessed by a specific thread. An index table for resource blocks is configured in the data shards. The index table records the resource block's identifier, physical storage location, shard information, and access permissions, allowing for quick location of the resource block's storage position. The status flags and index table are mapped to a data access channel. This channel defines the flow path and access method of resource data between the local cache layer and the global shared layer, generating a data access strategy. Data access strategies include read-write separation strategies, cache preheating strategies, data synchronization strategies, and expiration eviction strategies.

[0045] Based on the data access strategy, the local data cache layer is configured as a fast access queue, using in-memory storage to support high-concurrency read and write operations, with an average access latency of less than 1ms. The global data sharing layer is configured as a persistent queue, using distributed storage to support data persistence and consistency guarantees, with an average access latency of 10ms. Resource blocks are allocated to their corresponding queue levels, with the allocation principle being that high-access-frequency resource blocks are prioritized for the local data cache layer, while low-access-frequency but large-data-volume resource blocks are allocated to the global data sharing layer. Specifically, resource blocks accessed more than 5 times / s are allocated to the local data cache layer; resource blocks accessed less than 5 times / s are allocated to the global data sharing layer; for resource blocks accessed more than 5 times / s and with a data volume greater than 10KB, they are split into hot data and cold data, with hot data allocated to the local cache layer and cold data allocated to the global sharing layer. Queue structure data is generated, including queue hierarchy relationships, resource block allocation schemes, buffer configurations, status flag definitions, index table structures, access channel mappings, and other information.

[0046] In a distributed application system, there are mutually exclusive resource points A, B, C, D, and E. The memory address difference between A and B is 2KB, and they access each other 3 times / s; the memory address difference between B and C is 3KB, and they access each other 2.5 times / s; the memory address difference between C and D is 8KB, and they access each other 1 time / s; the memory address difference between D and E is 1KB, and they access each other 3.5 times / s. Based on a first threshold of 4KB and a second threshold of 2 times / s, A, B, and C are grouped into resource block 1, and D and E are grouped into resource block 2. Resource block 1 has a data capacity of 5KB and an access frequency of 7 times / s; resource block 2 has a data capacity of 3KB and an access frequency of 6 times / s. A two-layer distributed queue is constructed: the write buffer capacity of the local data cache layer is set to 20MB, and the read buffer capacity is set to 30MB; the data fragment size of the global data sharing layer is set to 2MB, and the transmission channel bandwidth is set to 10MB / s. Set resource block status flags in the write buffer and read buffer. Resource block 1 is flagged as "not dirty, not expired, unlocked," and resource block 2 is flagged as "dirty, not expired, locked." Configure an index table for the resource blocks in the data shards to record information such as the identifier, storage location, and access permissions of resource block 1 and resource block 2. Based on the data access strategy and allocation principles, allocate both resource block 1 and resource block 2 to the fast access queue of the local data cache layer, generating complete queue structure data.

[0047] This embodiment analyzes the spatial distribution and access relationships of mutually exclusive resource points in depth, organizes highly correlated resource points into resource blocks, and constructs a highly adaptable two-layer distributed queue, achieving efficient management and optimization of resource access. Through read-write separation design and a state marking mechanism, the method ensures data consistency and avoids access conflicts under high concurrency, effectively reducing thread contention overhead. It provides an intelligent resource management solution for high-concurrency software systems, balancing access speed, memory usage, and consistency requirements, and providing strong support for performance optimization in complex application scenarios.

[0048] The system monitors the execution status of the scheduling code, records thread blocking duration, and when a thread blocking duration exceeds a preset blocking time, splits the blocked thread into multiple sub-threads. These sub-threads are then allocated to idle computing nodes for parallel execution, generating parallel processing code including: Monitor the running status of the scheduling code, collect the number of thread switching and resource usage time, calculate the thread blocking duration, mark threads whose blocking duration exceeds the preset blocking time as blocked threads, and generate thread status data; Analyze the memory access sequence of blocked threads in the thread state data, divide the memory access sequence into continuous regions and discrete regions according to address continuity, construct computing tasks based on the continuous regions, construct distribution tasks based on the discrete regions, and generate a thread splitting scheme. Collect the processing performance and memory status of computing nodes, mark computing nodes with processing performance higher than a preset performance threshold and memory status greater than a preset memory threshold as idle computing nodes, and allocate sub-thread tasks to idle computing nodes based on the thread splitting scheme to generate a node allocation scheme. Based on the node allocation scheme, sub-thread tasks are deployed on idle computing nodes. Communication channels are established between sub-thread tasks to coordinate the parallel execution of sub-threads and generate parallel processing code.

[0049] The runtime status of the scheduling code is monitored, collecting data on thread switching counts and resource occupancy time. The thread switching count is collected by setting a counter in the scheduling code; the counter increments each time a thread context switch occurs, and the counter value is periodically read to obtain the thread switching frequency. Resource occupancy time is collected by recording the timestamps of a thread acquiring and releasing resources; the difference between these timestamps is the resource occupancy time. Thread blocking duration is calculated as the time interval from when a thread enters a waiting state to when it gets another chance to execute. Timestamps are recorded before and after the thread scheduling point; if the difference between the two timestamps is greater than the thread's normal execution cycle, the thread is considered blocked, and this difference is the blocking duration.

[0050] Threads whose blocking time exceeds a preset blocking time are marked as blocked threads. The preset blocking time is set to 100ms, meaning that a thread is marked as blocked when its blocking time exceeds 100ms. Thread state data is generated, including thread identifier, blocking state, blocking duration, blocking frequency, and blocking reason. The thread state data also includes thread execution history, such as CPU utilization, memory access patterns, and I / O operation frequency, for subsequent analysis and optimization.

[0051] This analysis examines the memory access sequences of blocked threads within the thread state data. A memory access sequence is a list of memory addresses accessed sequentially by a thread during execution, obtained using memory access tracing tools. The memory access sequences are divided into continuous and discrete regions based on address continuity. The criterion for address continuity is: if the offset between consecutively accessed memory addresses is less than 8 bytes, these addresses are considered to constitute a continuous region; otherwise, they constitute a discrete region. Computational tasks are constructed based on these continuous regions. A computational task is a unit of computation that performs intensive processing on a continuous memory region, characterized by computational intensity, strong data locality, and regular memory access patterns. The method for constructing computational tasks involves encapsulating related continuous memory operations into independent functions, setting input parameters and return values ​​to ensure the self-containment and independence of the tasks.

[0052] Based on discrete memory regions, a distribution task is constructed. A distribution task is a unit of work that performs data transmission and control coordination within discrete memory regions, characterized by I / O intensity, strong data dependencies, and high synchronization requirements. The method for constructing distribution tasks involves organizing memory operations in discrete regions into data flow pipelines to achieve asynchronous data transmission and processing. A thread splitting scheme is generated, including the splitting structure of the original blocking threads, the type of sub-thread tasks (computation tasks or distribution tasks), task dependencies, resource requirements, and other information. The thread splitting scheme also includes task scheduling strategies, such as task priority, scheduling order, and load balancing mechanisms.

[0053] The system collects data on the processing performance and memory status of compute nodes. Processing performance metrics include CPU utilization, instruction execution speed, and cache hit rate; memory status metrics include available memory capacity, memory bandwidth utilization, and page swapping frequency. Compute nodes with processing performance exceeding a preset performance threshold and memory status exceeding a preset memory threshold are marked as idle compute nodes. The preset performance threshold is set as CPU utilization below 30%, and the preset memory threshold is set as available memory capacity greater than 1GB. Compute nodes meeting both conditions are marked as idle compute nodes.

[0054] The thread-splitting scheme allocates sub-thread tasks to idle computing nodes. Task allocation employs an affinity scheduling strategy, meaning tasks with strong data dependencies are assigned to the same or neighboring nodes to reduce communication overhead. Load balancing is also considered to prevent some nodes from being overloaded while others are idle. A node allocation scheme is generated, including a list of sub-thread tasks deployed on each idle computing node, resource allocation strategies, and communication topology.

[0055] Sub-thread tasks are deployed on idle computing nodes according to the node allocation scheme. The deployment process includes task code transfer, runtime environment preparation, resource pre-allocation, and startup parameter settings. Communication channels are established between sub-thread tasks. These channels can be implemented using shared memory, message queues, network sockets, etc., with the most suitable communication mechanism selected based on task communication requirements and node distribution characteristics. The parallel execution of sub-threads is coordinated through the communication channels. Coordination mechanisms include data synchronization, task orchestration, exception handling, and result aggregation. Parallel processing code is generated, including sub-thread task code, communication coordination code, and scheduling control code, implementing a parallel optimized version of the blocking thread functionality.

[0056] If the blocking time of thread T1 reaches 350ms, exceeding the preset blocking time of 100ms, it is marked as a blocked thread, and thread state data is generated. Analyzing the memory access sequence of T1 reveals that it contains multiple continuous regions (C1, C2, C3) and discrete regions (D1, D2). Region C1 is 512KB in size and uses sequential read / write access; region C2 is 256KB in size and uses matrix calculation; region C3 is 128KB in size and uses vector operations; region D1 contains 23 scattered addresses, mainly for configuration data reading; region D2 contains 47 scattered addresses, mainly for result collection. Based on C1, C2, and C3, computation tasks T1-C1, T1-C2, and T1-C3 are constructed; based on D1 and D2, distribution tasks T1-D1 and T1-D2 are constructed, generating a thread splitting scheme.

[0057] The processing performance and memory status of computing nodes N1, N2, N3, and N4 are collected. N1 has a CPU utilization of 25% and 1.5GB of available memory; N2 has a CPU utilization of 40% and 2GB of available memory; N3 has a CPU utilization of 20% and 1.2GB of available memory; and N4 has a CPU utilization of 60% and 0.8GB of available memory. Based on preset thresholds, N1 and N3 are marked as idle computing nodes. Using a thread splitting scheme, T1-C1 and T1-D1 are assigned to N1, and T1-C2, T1-C3, and T1-D2 are assigned to N3, generating a node allocation scheme. According to the node allocation scheme, corresponding sub-thread tasks are deployed on N1 and N3, a communication channel based on shared memory is established, the parallel execution of sub-threads is coordinated, and parallel processing code containing task code and communication logic is generated.

[0058] This embodiment effectively solves the performance bottleneck problem caused by thread blocking in high-performance computing environments through intelligent monitoring and adaptive task splitting. It can accurately identify blocked threads in the system, analyze their memory access patterns, and rationally split tasks into computationally intensive and data-distributed types based on address continuity characteristics, fully utilizing the parallel processing capabilities of modern computing architectures. By collecting the resource status of computing nodes in real time, it can identify idle computing resources in the system and intelligently allocate the split subtasks to suitable nodes for execution, achieving optimal utilization of computing resources. The communication channel design between sub-threads ensures efficient and consistent data flow, making collaboration during parallel execution smoother.

[0059] Analyze the memory access sequences of blocked threads in the thread state data, divide the memory access sequences into continuous and discrete regions according to address continuity, construct computational tasks based on the continuous regions, and construct distribution tasks based on the discrete regions, generating a thread splitting scheme including: Analyze the memory access operations of blocked threads in the thread state data, obtain the execution order of the memory access operations, calculate the correlation between memory access operations based on the execution order, and generate a memory access sequence; The correlation degree of memory access operations in the memory access sequence is analyzed. Memory access operations with a correlation degree higher than a preset correlation threshold are divided into continuous regions, and memory access operations with a correlation degree lower than the preset correlation threshold are divided into discrete regions, generating region division information. Extract consecutive memory access operations from contiguous regions, organize the pipeline structure based on the execution order of consecutive memory access operations, build data transmission channels in the pipeline structure, and generate computation tasks. Extract discrete memory access operations from discrete regions, organize the cache structure based on the execution order of discrete memory access operations, build data exchange channels in the cache structure, and generate distribution tasks. The data transmission channel of the computing task is connected to the data exchange channel of the distribution task to form a data path, and a thread splitting scheme is generated based on the data path.

[0060] Memory access operations refer to the basic operations performed by a thread during execution, including read, write, and atomic operations. The execution order is obtained by inserting memory access tracking points during program execution, recording the timestamp, memory address, and operation type of each memory access operation, forming a time-series memory access log. The correlation between memory access operations is calculated based on the execution order, using two dimensions: temporal locality and spatial locality. Temporal locality refers to the temporal proximity of two memory access operations, calculated through timestamp differences; spatial locality refers to the proximity of two memory access operations in their memory address space, calculated through memory address differences. A weighted sum of temporal and spatial localities is then used to obtain the correlation between memory access operations, generating a memory access sequence.

[0061] The correlation between memory access operations in the memory access sequence is analyzed. Memory access operations with a correlation higher than a preset correlation threshold are classified into continuous regions, while those with a correlation lower than the preset threshold are classified into discrete regions. The preset correlation threshold is determined based on system performance characteristics and application scenario requirements, and generally ranges from 0.6 to 0.8. Region partitioning information is generated, including boundary markers for continuous and discrete regions, memory address ranges, and access pattern characteristics. The region partitioning information also includes a list of memory access operations within each region and their correlation diagram, providing a basis for subsequent task construction.

[0062] Extracting consecutive memory access operations from contiguous regions, a pipeline structure is organized based on the execution order of these operations. A pipeline structure is a parallel computing model that divides consecutive memory access operations into multiple execution stages, each of which can be executed in parallel, improving processing efficiency. The pipeline structure is organized by dividing consecutive memory access operations into multiple stages according to data dependencies, ensuring smooth data flow between stages. Data transmission channels are constructed within the pipeline structure, serving as data pathways connecting the different stages and used to transmit intermediate computation results. Implementation methods for data transmission channels include shared memory regions, inter-thread message queues, and pipes; the appropriate implementation method is selected based on the data transmission volume and real-time requirements. Computational tasks are generated. These tasks encapsulate consecutive memory access operations, possessing clearly defined input, processing logic, and output, and can be deployed as independent execution units on computing resources.

[0063] Discrete memory access operations in discrete regions are extracted, and the cache structure is organized based on the execution order of these operations. The cache structure is a data storage and management model that employs a multi-level caching strategy to reduce memory access latency, taking into account the characteristics of discrete memory access. The cache structure is organized by layering discrete memory access operations according to access frequency and pattern characteristics, placing high-frequency access data in the first-level cache and low-frequency access data in the second-level cache. A data exchange channel is constructed within the cache structure, serving as a data path connecting the various cache levels for data loading, updating, and synchronization. Implementation methods for the data exchange channel include asynchronous prefetching mechanisms, write-back strategies, and cache consistency protocols; the appropriate implementation method is selected based on data consistency requirements and system performance goals. Distribution tasks are generated; these tasks encapsulate the discrete region memory access operations and are responsible for data distribution, collection, and coordination, functioning as independent execution units that work collaboratively with computational tasks.

[0064] A data path is formed by connecting the data transmission channel of the computation task with the data exchange channel of the distribution task. The data path is the data flow path that runs through the entire thread execution flow, connecting the computation task and the distribution task to achieve efficient data transfer and sharing. The connection method of the data path involves establishing a data mapping relationship between the computation task and the distribution task, defining data format conversion rules, and ensuring the compatibility and consistency of data flow. Based on the data path, a thread splitting scheme is generated. The thread splitting scheme includes sub-thread partitioning structure, task allocation strategy, data flow mechanism, synchronization control method, etc., providing a complete solution for subsequent parallelization implementation.

[0065] For example, consider a blocked thread T. Its memory access operation sequence, consisting of 128 memory access operations numbered Op1 to Op128, is obtained through memory access tracing. The correlation between these memory access operations is calculated. It is found that the correlation coefficients of operations Op1 to Op42 are mostly above 0.85, those of operations Op43 to Op78 are mostly below 0.35, and those of operations Op79 to Op128 are mostly above 0.90. Based on a preset correlation threshold of 0.75, Op1 to Op42 are divided into a continuous region R1, Op43 to Op78 into a discrete region R2, and Op79 to Op128 into a continuous region R3, generating region partitioning information.

[0066] Memory access operations in the continuous region R1 are extracted and, after analyzing the execution order, organized into a three-stage pipeline structure: data loading, feature extraction, and data transformation. A shared memory-based data transfer channel is constructed within this pipeline structure, with data passed between stages via memory buffers, generating computation task T1. Memory access operations in the continuous region R3 are extracted and, after analyzing the execution order, organized into a four-stage pipeline structure: data filtering, aggregation calculation, sorting, and result output. A message queue-based data transfer channel is constructed within this pipeline structure, with data passed between stages via messages, generating computation task T3. Memory access operations in the discrete region R2 are extracted and, after analyzing the execution order, organized into a two-level cache structure. The first-level cache stores frequently accessed data with a capacity of 64KB, while the second-level cache stores infrequently accessed data with a capacity of 256KB. An asynchronous prefetch-based data exchange channel is constructed within this cache structure, employing a write-back strategy to ensure data consistency, generating dispatch task T2. The data transmission channel of computation task T1 is connected to the data exchange channel of distribution task T2 through a data conversion interface, and the data exchange channel of distribution task T2 is connected to the data transmission channel of computation task T3 through a data synchronization interface, forming a complete data path. A thread splitting scheme is generated based on this data path. The thread splitting scheme divides the original blocked thread T into three sub-threads, which execute computation task T1, distribution task T2, and computation task T3 respectively, achieving data flow and task coordination through the defined data path.

[0067] This embodiment achieves fine-grained task decomposition and resource scheduling through in-depth analysis of thread memory access characteristics, effectively solving the performance bottleneck problem caused by neglecting differences in memory access patterns in traditional parallelization methods. This method can accurately identify the correlation between memory access operations, organizing highly correlated operations into a continuous computational structure and optimizing low-correlation operations into a distributed cache structure, fully utilizing the caching mechanisms and parallel processing capabilities of modern processors. By managing discrete memory access through multi-level caching, it effectively reduces memory access conflicts and cache miss rates.

[0068] The execution results of parallel processing code are collected, the response time and resource utilization of sub-threads are calculated, and the execution path with the shortest response time and highest resource utilization is recorded as a scheduling template. Based on the scheduling template, the system resource allocation strategy is determined, and a software development plan is generated, including: Collect the execution results of parallel processing code, record the start and end times of sub-threads, monitor the memory usage and processor utilization of sub-threads, calculate the runtime based on the time difference, calculate the resource consumption value based on usage and utilization, and combine the runtime and resource consumption value to generate performance index data. The runtime in the performance metrics data is marked as response time, and the resource consumption value is marked as resource utilization rate. A two-dimensional performance evaluation matrix is ​​constructed based on the response time and resource utilization rate. The execution paths in the two-dimensional performance evaluation matrix are sorted to generate path analysis data. Extract the execution path with the shortest response time and highest resource utilization from the path analysis data, record the thread allocation method and resource configuration parameters of the execution path, record the execution path as a scheduling template, and generate scheduling template data; The scheduling template data is converted into a system resource allocation strategy. Based on the system resource allocation strategy, thread priorities and resource quotas are set to generate a software development plan.

[0069] The system collects the execution results of parallel processing code, recording the start and end times of child threads. The start time refers to the point in time when the child thread is created and begins execution, obtained by acquiring the system time immediately after the thread creation function. The end time refers to the point in time when the child thread completes all tasks and exits, obtained by acquiring the system time before the thread terminates. The system monitors the memory usage and processor utilization of child threads. Memory usage is monitored by periodically collecting thread memory allocation and release information using memory analysis tools, recording peak and average memory usage. Processor utilization is monitored by periodically collecting thread CPU time slice usage using performance counters, calculating CPU utilization. The runtime is calculated based on the time difference, equal to the end time minus the start time, representing the total execution time of the child thread from creation to termination. Resource consumption is calculated based on usage and utilization, using a weighted summation method. Memory usage and processor utilization are combined according to configured weights, determined based on system characteristics and application scenarios; typically, memory usage has a weight of 0.4, and processor utilization has a weight of 0.6. The runtime and resource consumption values ​​are combined to generate performance metric data. The performance metric data includes information such as the identifier, runtime, and resource consumption value of each sub-thread, which constitutes the basic dataset for performance evaluation.

[0070] Runtime in the performance metrics data is labeled as response time, and resource consumption is labeled as resource utilization. Response time is a key indicator of software execution efficiency, reflecting user experience and system real-time performance; resource utilization is a key indicator of software resource usage efficiency, reflecting the effectiveness of system resource utilization. A two-dimensional performance evaluation matrix is ​​constructed based on response time and resource utilization. Rows in the matrix represent different sub-threads or execution units, and columns represent the two dimensions of response time and resource utilization. Matrix elements contain corresponding performance data. The execution paths in the two-dimensional performance evaluation matrix are sorted. An execution path refers to a complete processing flow from input to output in parallel processing code, containing a collaborative sequence of multiple sub-threads. First, they are sorted by response time from smallest to largest; if response times are the same, they are then sorted by resource utilization from highest to lowest, forming a priority ranking list. Path analysis data is generated, including the sorted execution path list, performance metrics for each path, and information on key sub-threads within each path.

[0071] The execution path with the shortest response time and highest resource utilization is extracted from the path analysis data. This execution path represents the optimal processing flow in the current parallel processing code. The thread allocation method and resource configuration parameters of this execution path are recorded. The thread allocation method includes the number of child threads, the binding relationship between child threads and processor cores, and the synchronization mechanism between child threads. The resource configuration parameters include memory allocation strategy, cache usage strategy, and I / O buffer size. This execution path is recorded as a scheduling template, generating scheduling template data. The scheduling template data includes template identifier, applicable conditions, thread allocation method, resource configuration parameters, performance expectations, etc., for easy subsequent querying and application.

[0072] The scheduling template data is converted into a resource allocation policy. The source allocation policy is a resource management instruction set oriented towards the operating system, including processor scheduling policies, memory allocation policies, and I / O scheduling policies. The conversion method maps the thread allocation method in the scheduling template to processor affinity settings and the resource configuration parameters to resource limits and reserved configurations. Thread priorities and resource quotas are set according to the resource allocation policy. Thread priorities follow the critical path priority principle, meaning that sub-threads with a greater impact on overall performance in the execution path receive higher priority. Resource quotas follow the on-demand allocation principle, meaning that appropriate resources are allocated based on the resource requirements of sub-threads. A software development plan is generated, including an optimized code structure, resource configuration files, deployment instructions, performance test reports, etc., providing complete guidance for software development and deployment.

[0073] In a data processing application, parallel processing code contains five sub-threads: T1, T2, T3, T4, and T5. Execution results were collected and the start and end times of each thread were recorded. T1 started at 0ms and ended at 120ms; T2 started at 10ms and ended at 90ms; T3 started at 30ms and ended at 150ms; T4 started at 50ms and ended at 180ms; and T5 started at 100ms and ended at 200ms. Monitoring revealed the following memory usage and processor utilization for each thread: T1's average memory usage was 256MB, and its processor utilization was 45%; T2's average memory usage was 128MB, and its processor utilization was 70%; T3's average memory usage was 512MB, and its processor utilization was 30%; T4's average memory usage was 320MB, and its processor utilization was 60%; and T5's average memory usage was 96MB, and its processor utilization was 25%. The runtime of each thread was calculated as follows: T1 = 120ms, T2 = 80ms, T3 = 120ms, T4 = 130ms, and T5 = 100ms. Resource consumption was calculated based on the configured weights: T1 = 0.4×256 + 0.6×45 = 148.2, T2 = 0.4×128 + 0.6×70 = 93.2, T3 = 0.4×512 + 0.6×30 = 222.8, T4 = 0.4×320 + 0.6×60 = 164, and T5 = 0.4×96 + 0.6×25 = 53.4. Performance metrics data were generated, including the runtime and resource consumption of each thread.

[0074] Runtime is labeled as response time, and resource consumption is labeled as resource utilization, constructing a two-dimensional performance evaluation matrix. Analyzing the execution paths, three main execution paths were identified: Path P1, containing threads T1→T3→T5, has a total response time of 340ms and an average resource utilization of 141.47; Path P2, containing threads T2→T4→T5, has a total response time of 310ms and an average resource utilization of 103.53; Path P3, containing threads T1→T2→T4, has a total response time of 330ms and an average resource utilization of 135.13. Ranking the execution paths, path P2 has the shortest response time, while path P1 has the highest resource utilization. Considering all factors, path P2 is selected as the optimal execution path. Record the thread allocation method for path P2: T2 is bound to CPU cores 1 and 2, T4 is bound to CPU cores 3 and 4, and T5 is bound to CPU core 5; resource configuration parameters: T2 memory limit is 256MB, T4 memory limit is 512MB, and T5 memory limit is 128MB. Record path P2 as a scheduling template, generate scheduling template data, including template identifier "FastDataProcess", applicable conditions "medium-sized data processing task", etc. Convert the scheduling template data into a system resource allocation policy, setting the thread priority of T2 to 90, T4 priority to 80, and T5 priority to 70; setting the CPU time slice quota of T2 to 60%, T4 quota to 30%, and T5 quota to 10%; setting memory resource limits and disk I / O bandwidth allocation. Generate a software development plan, including optimized parallel processing code, resource configuration files, and deployment instructions.

[0075] This embodiment achieves deep optimization and adaptive scheduling of parallel processing code through refined performance data collection and multi-dimensional performance index analysis. It accurately captures the time consumption and resource utilization during software operation, constructs a two-dimensional performance evaluation matrix, and extracts the optimal execution path, effectively solving the performance bottleneck problem caused by unreasonable resource allocation in traditional development methods. By refining the optimal execution path into a scheduling template and transforming it into a system-level resource allocation strategy, this method establishes a closed-loop optimization mechanism from performance analysis to resource scheduling, enabling the software system to dynamically adjust resource configuration according to actual operating conditions.

[0076] like Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a software development solution generation system, the system comprising: The data acquisition module is used to collect system resource usage data and thread execution data during multi-user concurrent operations. It divides the system resource usage data into resource segments according to the time dimension, locates resource contention points in the resource segments, and associates the resource contention points with the thread execution data to generate conflict marker data. The scheduling processing module is used to construct a thread execution graph based on conflict marker data, identify mutual exclusion resource points in the thread execution graph, construct a two-level distributed queue based on the mutual exclusion resource points, set resource status listeners in the two-level distributed queue, dynamically allocate thread priorities based on the resource status listeners, and generate scheduling processing code. The parallel processing module is used to monitor the running status of the scheduling processing code, record the thread blocking time, and when the thread blocking time is detected to exceed the preset blocking time, the blocked thread is split into multiple sub-threads, and the sub-threads are allocated to idle computing nodes for parallel execution to generate parallel processing code. The scheme generation module is used to collect the execution results of parallel processing code, calculate the response time and resource utilization of sub-threads, record the execution path with the shortest response time and the highest resource utilization as a scheduling template, determine the system resource allocation strategy based on the scheduling template, and generate a software development scheme.

[0077] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0078] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating a software development solution, characterized in that, Includes the following steps: Collect system resource usage data and thread execution data during multi-user concurrent operations, divide the system resource usage data into resource segments according to the time dimension, locate resource contention points in the resource segments, and associate the resource contention points with the thread execution data to generate conflict marker data; A thread execution graph is constructed based on conflict marker data. Mutual exclusive resource points are identified in the thread execution graph. A two-layer distributed queue is constructed based on the mutual exclusive resource points. Resource status listeners are set in the two-layer distributed queue. Thread priorities are dynamically allocated based on the resource status listeners, and scheduling processing code is generated. Monitor the running status of the scheduling code, record the thread blocking time, and when the thread blocking time exceeds the preset blocking time, split the blocked thread into multiple sub-threads, allocate the sub-threads to idle computing nodes for parallel execution, and generate parallel processing code. The execution results of parallel processing code are collected, the response time and resource utilization of sub-threads are calculated, the execution path with the shortest response time and the highest resource utilization is recorded as a scheduling template, the system resource allocation strategy is determined based on the scheduling template, and a software development plan is generated.

2. The method according to claim 1, characterized in that, Collect system resource usage data and thread execution data during multi-user concurrent operations. Divide the system resource usage data into resource segments based on time. Locate resource contention points within these segments and correlate these points with thread execution data to generate conflict marker data, including: Collect system resource usage data and thread execution data, calculate resource utilization rate from the system resource usage data, generate resource utilization curves according to the collection time sequence, and divide resource segments based on the fluctuation trend of the resource utilization curves; Identify abrupt changes in the resource usage curve within a resource segment, mark these abrupt changes as resource contention points, extract thread execution data at the time of the resource contention point into thread combinations, calculate the access interval of each thread in the thread combination to the same resource, and generate access time sequence data. The overlapping intervals of thread access in the access time sequence data are marked, the resource occupation time of the threads in the overlapping interval is calculated, and when the resource occupation time is longer than the preset occupation time, the corresponding thread group is marked as a competition group and conflict marking data is generated.

3. The method according to claim 1, characterized in that, A thread execution graph is constructed based on conflict marker data. Mutual exclusive resource points are identified within the graph. A two-layer distributed queue is built based on these points. Resource status listeners are set within the queues. Thread priorities are dynamically allocated based on these listeners. Scheduling code is then generated, including: A thread execution graph is constructed based on conflict marker data. Threads in the thread execution graph are set as nodes, and resource access relationships between threads are set as connecting edges. The resource contention value of the connecting edges is calculated, and thread association data is generated. The resource access frequency of nodes is calculated based on thread-related data. Nodes whose resource access frequency exceeds a preset frequency value are marked as mutual exclusion resource points. A two-layer distributed queue is constructed based on the mutual exclusion resource points. A local data caching layer and a global data sharing layer are set in the two-layer distributed queue to generate queue structure data. Based on the queue structure data, resource status listeners are set up in the local data cache layer and the global data sharing layer respectively to monitor the length of the resource queue. When the queue length exceeds the preset length value, the resource access request is allocated to the local data cache layer. When the resource processing time exceeds the preset time value, the data is updated to the global data sharing layer to generate resource status data. Based on the resource status data collected in real time by the resource status listener, the ratio of resource occupancy time to preset base time is calculated periodically, the thread priority is dynamically updated according to the ratio, and scheduling processing code is generated based on the updated thread priority.

4. The method according to claim 3, characterized in that, A two-layer distributed queue is constructed based on mutually exclusive resource points. A local data caching layer and a global data sharing layer are set up within the two-layer distributed queue. The generated queue structure data includes: The spatial distribution and access relationships between mutually exclusive resource points are statistically analyzed. Mutually exclusive resource points whose spatial distance is less than a first threshold and whose access frequency is greater than a second threshold are grouped into resource blocks. The data capacity and access popularity of the resource blocks are calculated to generate resource distribution data. A two-layer distributed queue is constructed based on resource distribution data. A local data cache layer and a global data sharing layer are set in the two-layer distributed queue. The local data cache layer is divided into a write buffer and a read buffer. Data fragmentation and transmission channels are established in the global data sharing layer. Set the status flag of the resource block in the write buffer and read buffer, configure the index table of the resource block in the data fragment, map the status flag and index table into a data access channel, and generate a data access strategy. According to the data access strategy, the local data cache layer is configured as a fast access queue, the global data sharing layer is configured as a persistent queue, the resource blocks are allocated to the corresponding queue levels, and queue structure data is generated.

5. The method according to claim 1, characterized in that, The system monitors the execution status of the scheduling code, records thread blocking duration, and when a thread blocking duration exceeds a preset blocking time, splits the blocked thread into multiple sub-threads. These sub-threads are then allocated to idle computing nodes for parallel execution, generating parallel processing code including: Monitor the running status of the scheduling code, collect the number of thread switching and resource usage time, calculate the thread blocking duration, mark threads whose blocking duration exceeds the preset blocking time as blocked threads, and generate thread status data; Analyze the memory access sequence of blocked threads in the thread state data, divide the memory access sequence into continuous regions and discrete regions according to address continuity, construct computing tasks based on the continuous regions, construct distribution tasks based on the discrete regions, and generate a thread splitting scheme. Collect the processing performance and memory status of computing nodes, mark computing nodes with processing performance higher than a preset performance threshold and memory status greater than a preset memory threshold as idle computing nodes, and allocate sub-thread tasks to idle computing nodes based on the thread splitting scheme to generate a node allocation scheme. Based on the node allocation scheme, sub-thread tasks are deployed on idle computing nodes. Communication channels are established between sub-thread tasks to coordinate the parallel execution of sub-threads and generate parallel processing code.

6. The method according to claim 5, characterized in that, Analyze the memory access sequences of blocked threads in the thread state data, divide the memory access sequences into continuous and discrete regions according to address continuity, construct computational tasks based on the continuous regions, and construct distribution tasks based on the discrete regions, generating a thread splitting scheme including: Analyze the memory access operations of blocked threads in the thread state data, obtain the execution order of the memory access operations, calculate the correlation between memory access operations based on the execution order, and generate a memory access sequence; The correlation degree of memory access operations in the memory access sequence is analyzed. Memory access operations with a correlation degree higher than a preset correlation threshold are divided into continuous regions, and memory access operations with a correlation degree lower than the preset correlation threshold are divided into discrete regions, generating region division information. Extract consecutive memory access operations from contiguous regions, organize the pipeline structure based on the execution order of consecutive memory access operations, build data transmission channels in the pipeline structure, and generate computation tasks. Extract discrete memory access operations from discrete regions, organize the cache structure based on the execution order of discrete memory access operations, build data exchange channels in the cache structure, and generate distribution tasks. The data transmission channel of the computing task is connected to the data exchange channel of the distribution task to form a data path, and a thread splitting scheme is generated based on the data path.

7. The method according to claim 1, characterized in that, The execution results of parallel processing code are collected, the response time and resource utilization of sub-threads are calculated, and the execution path with the shortest response time and highest resource utilization is recorded as a scheduling template. Based on the scheduling template, the system resource allocation strategy is determined, and a software development plan is generated, including: Collect the execution results of parallel processing code, record the start and end times of sub-threads, monitor the memory usage and processor utilization of sub-threads, calculate the runtime based on the time difference, calculate the resource consumption value based on usage and utilization, and combine the runtime and resource consumption value to generate performance index data. The runtime in the performance metrics data is marked as response time, and the resource consumption value is marked as resource utilization rate. A two-dimensional performance evaluation matrix is ​​constructed based on the response time and resource utilization rate. The execution paths in the two-dimensional performance evaluation matrix are sorted to generate path analysis data. Extract the execution path with the shortest response time and highest resource utilization from the path analysis data, record the thread allocation method and resource configuration parameters of the execution path, record the execution path as a scheduling template, and generate scheduling template data; The scheduling template data is converted into a system resource allocation strategy. Based on the system resource allocation strategy, thread priorities and resource quotas are set to generate a software development plan.

8. A software development scheme generation system, used to implement the method described in any one of claims 1-7, characterized in that, The system includes: The data acquisition module is used to collect system resource usage data and thread execution data during multi-user concurrent operations. It divides the system resource usage data into resource segments according to the time dimension, locates resource contention points in the resource segments, and associates the resource contention points with the thread execution data to generate conflict marker data. The scheduling processing module is used to construct a thread execution graph based on conflict marker data, identify mutual exclusion resource points in the thread execution graph, construct a two-level distributed queue based on the mutual exclusion resource points, set resource status listeners in the two-level distributed queue, dynamically allocate thread priorities based on the resource status listeners, and generate scheduling processing code. The parallel processing module is used to monitor the running status of the scheduling processing code, record the thread blocking time, and when the thread blocking time is detected to exceed the preset blocking time, the blocked thread is split into multiple sub-threads, and the sub-threads are allocated to idle computing nodes for parallel execution to generate parallel processing code. The scheme generation module is used to collect the execution results of parallel processing code, calculate the response time and resource utilization of sub-threads, record the execution path with the shortest response time and the highest resource utilization as a scheduling template, determine the system resource allocation strategy based on the scheduling template, and generate a software development scheme.

Images