A solid state memory data processing method and system

Abstract

The application relates to the technical field of solid-state memory data processing, and discloses a solid-state memory data processing method and system. The method comprises the following steps: collecting a real-time operation sequence, performing semantic analysis on the operation sequence, and obtaining an operation semantic representation; performing statistical analysis and priority sorting according to the operation semantic representation, and obtaining a hierarchical task queue; dynamically adjusting a storage resource of the hierarchical task queue, and obtaining a resource distribution structure; determining a potential resource bottleneck position according to the resource distribution structure, and obtaining a storage space mapping; determining a data access channel according to the storage space mapping; detecting multi-thread conflicts in the data access channel and adjusting the multi-thread conflicts, and obtaining a response time distribution; dynamically updating semantic parameters and limiting an adjustment range according to the response time distribution, and obtaining a data processing instruction. The method can solve the problem that the overall system performance bottleneck is prominent under a complex dynamic workload in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of solid state memory data processing, and particularly relates to a solid state memory data processing method and system. BACKGROUND

[0002] At present, in the field of modern information technology, as the core storage component of computing devices, the data processing efficiency of solid state memory directly affects the system performance and user experience. The data management method, as the core of performance optimization of solid state memory, directly determines the running efficiency of the overall storage system in terms of adaptability to dynamic working environment and rationality of resource allocation.

[0003] In one prior art, the solid state memory data processing adopts a fixed resource allocation and preset process scheduling method. First, the buffer allocation ratio and data arrangement priority are set according to historical experience. After collecting input and output requests through the host interface, the data storage and access scheduling are performed according to the fixed process. Finally, the internal data arrangement and space recycling are completed based on the preset rules. However, the prior art does not consider the dynamic changes of the host operation intention. In the face of the difference in urgency of different tasks in the multi-task concurrent scene, the fixed resource allocation cannot guarantee the resource demand of high-priority tasks, and the preset scheduling process is difficult to adapt to the fluctuation of data access frequency, resulting in access delay of high-frequency access data blocks due to insufficient buffer.

[0004] In summary, the prior art has the problem of prominent system overall performance bottleneck under complex dynamic workloads. SUMMARY

[0005] The present application provides a solid state memory data processing method and system to solve the problem of prominent system overall performance bottleneck under complex dynamic workloads in the prior art.

[0006] In a first aspect, to solve the above technical problems, the present application provides a solid state memory data processing method, comprising:

[0007] Collecting real-time operation sequences, performing semantic analysis on the real-time operation sequences to obtain operation semantic representations;

[0008] According to the operation semantic representations, performing statistical analysis and priority sorting to obtain a hierarchical task queue;

[0009] Performing dynamic adjustment of storage resources on the hierarchical task queue to obtain a resource distribution structure;

[0010] According to the resource distribution structure, performing space recycling operation and monitoring resource utilization to determine the potential resource bottleneck position, and obtaining a storage space mapping;

[0011] According to the storage space mapping, an update cache access path is determined, and a data access channel is determined;

[0012] Multi-thread conflict in the data access channel is detected and adjusted to obtain a response time distribution;

[0013] According to the response time distribution, resource utilization is tracked, semantic parameters are dynamically updated, and the adjustment range is limited to obtain a data processing instruction.

[0014] In an optional implementation, a real-time operation sequence is collected, and semantic analysis is performed on the real-time operation sequence to obtain an operation semantic representation, including:

[0015] A real-time operation sequence is collected, and the real-time operation sequence is segmented and processed to extract keywords and context information to obtain semantic features;

[0016] According to the semantic features, a multi-dimensional semantic vector is constructed, and task type and urgency features in the multi-dimensional semantic vector are weighted to determine a weighted semantic vector;

[0017] The weighted semantic vector is compared with a preset template library, and a matching template is fused with the semantic features to obtain an operation semantic representation.

[0018] In an optional implementation, according to the operation semantic representation, statistical analysis and priority sorting are performed to obtain a hierarchical task queue, including:

[0019] According to the operation semantic representation, data block access frequency and operation sequence mode are counted to obtain an operation behavior feature set;

[0020] The urgency and dependency of tasks in the operation behavior feature set are compared, and if the urgency exceeds a preset urgency threshold, operation parameters are adjusted to determine a task list;

[0021] The task list is adjusted and scheduled in layers to obtain a hierarchical task queue.

[0022] In an optional implementation, the hierarchical task queue is dynamically adjusted in storage resources to obtain a resource distribution structure, including:

[0023] Based on a preset proportion threshold, the hierarchical task queue is divided into tasks, and the data block access frequency is analyzed for relevance to determine a data block set;

[0024] According to the data block set, buffer allocation is optimized and hierarchical management is performed to obtain a resource distribution structure.

[0025] In an optional implementation, the performing a space recycling operation and monitoring resource utilization according to the resource distribution structure, determining a potential resource bottleneck position, and obtaining a storage space mapping, comprises:

[0026] Based on the resource distribution structure, extracting a task-associated data block satisfying a preset judgment condition and performing priority sorting, to obtain a to-be-processed data block set;

[0027] Reordering a data block storage location in the to-be-processed data block set, and determining a data block distribution condition;

[0028] If the data block distribution condition does not reach a preset distribution standard, analyzing a storage resource occupation proportion corresponding to the resource distribution structure, and determining a potential resource bottleneck position;

[0029] According to the potential resource bottleneck position, dynamically adjusting the data block storage location, and obtaining a storage space mapping.

[0030] In an optional implementation, the updating a cache access path according to the storage space mapping, and determining a data access channel, comprises:

[0031] According to the storage space mapping, performing path planning, and determining a path allocation scheme;

[0032] Performing time smoothing processing on the path allocation scheme, obtaining a smoothed fluctuation range, and determining a delay tolerance;

[0033] If the delay tolerance does not reach a preset tolerance threshold, adjusting the cache access path, and determining the data access channel.

[0034] In an optional implementation, the detecting a multi-thread conflict in the data access channel and adjusting, and obtaining a response time distribution, comprises:

[0035] If the data access channel has a multi-thread conflict, obtaining a detailed position of a conflict node, adjusting a storage resource allocation priority corresponding to the resource distribution structure according to the detailed position of the conflict node, and obtaining a resource distribution scheme;

[0036] If the resource distribution scheme cannot meet a demand of a concurrent request, adjusting the storage resource allocation priority, and obtaining a request processing queue;

[0037] Performing time smoothing processing on the request processing queue, and obtaining a response time distribution.

[0038] In an optional implementation, the tracking resource utilization according to the response time distribution, dynamically updating a semantic parameter and limiting an adjustment amplitude, and obtaining a data processing instruction, comprises:

[0039] According to the response time distribution, the resource utilization rate is tracked, a specific period of resource occupation peak is obtained, the update amplitude of the semantic parameter is processed in stages for the specific period, the parameter adjustment frequency is limited if the update amplitude exceeds a preset amplitude threshold, and a parameter configuration scheme is obtained;

[0040] According to the parameter configuration scheme, the priority of the semantic parameter is sorted, and a parameter application order is determined;

[0041] According to the parameter application order, the processing demand of the real-time operation sequence is loaded step by step, and a data processing instruction is obtained.

[0042] In a second aspect, the present application provides a solid-state memory data processing system, comprising:

[0043] An operation semantic capture module collects a real-time operation sequence, performs semantic analysis on the real-time operation sequence, and obtains an operation semantic representation;

[0044] A hierarchical task queue generation module performs statistical analysis and priority sorting according to the operation semantic representation, and obtains a hierarchical task queue;

[0045] A resource distribution adjustment module performs dynamic adjustment of storage resources on the hierarchical task queue, and obtains a resource distribution structure;

[0046] A storage space mapping optimization module performs a space recycling operation and monitors resource utilization rate according to the resource distribution structure, determines a potential resource bottleneck position, and obtains a storage space mapping;

[0047] A data access channel determination module updates a cache access path and determines a data access channel according to the storage space mapping;

[0048] A concurrent conflict coordination module detects and adjusts multi-thread conflicts in the data access channel, and obtains a response time distribution;

[0049] A flow refining module tracks resource utilization rate according to the response time distribution, dynamically updates semantic parameters and limits adjustment amplitude, and obtains a data processing instruction.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] (1) The application synchronously collects real-time operation sequences and input and output requests through a host interface, constructs a multi-dimensional semantic vector containing task type and emergency degree characteristics using semantic analysis technology, and generates a preliminary operation semantic representation by comparing with a preset template library, thereby providing accurate intention basis for subsequent task layering and resource allocation, and solving the problem of low priority tasks occupying resources and system response delay caused by the lack of deep analysis of operation semantics and the reliance on a single index to determine task priority in the prior art.

[0052] (2) The application adjusts storage resources dynamically according to high proportion tasks in the layered task queue, optimizes buffer allocation by analyzing data block correlation, and identifies bottlenecks and adjusts data block storage location in the space recycling phase to obtain an optimized storage space mapping, thereby ensuring that storage resources are tilted to high demand tasks and solving the problem of low storage resource utilization and data access congestion caused by the fixed resource allocation mode of the prior art, which cannot adapt to dynamic task load.

[0053] (3) The application adjusts the resource allocation order based on the data block access frequency and task emergency degree in the resource distribution structure when multi-thread conflicts are detected in the data access channel, combines the feedback loop data sorting request queue and performs time smoothing processing to obtain a smooth system response time distribution, thereby achieving active and accurate processing of concurrent conflicts and solving the problem of large system response fluctuation and poor stability under high load caused by the random avoidance strategy of the prior art, which cannot guarantee the priority processing of critical tasks. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a solid state memory data processing method flow chart provided by the first embodiment of the application;

[0055] Figure 2 is a solid state memory data processing system structure diagram provided by the second embodiment of the application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0057] Referring to Figure 1 The first embodiment of the application provides a solid state memory data processing method, including the following steps:

[0058] S1, collect real-time operation sequences, perform semantic analysis on the real-time operation sequences to obtain operation semantic representations;

[0059] S2, perform statistical analysis and priority sorting according to the operation semantic representations to obtain a hierarchical task queue;

[0060] S3, perform dynamic adjustment of storage resources on the hierarchical task queue to obtain a resource distribution structure;

[0061] S4, perform a space recycling operation and monitor resource utilization according to the resource distribution structure to determine a potential resource bottleneck position to obtain a storage space mapping;

[0062] S5, update a cache access path according to the storage space mapping to determine a data access channel;

[0063] S6, detect multi-thread conflicts in the data access channel and perform adjustment to obtain a response time distribution;

[0064] S7, track resource utilization according to the response time distribution, dynamically update semantic parameters and limit the adjustment range to obtain data processing instructions.

[0065] In step S1, real-time operation sequences are collected, and semantic analysis is performed on the real-time operation sequences to obtain operation semantic representations, including:

[0066] The real-time operation sequences are collected, and the real-time operation sequences are segmented and processed to extract keywords and context information to obtain semantic features;

[0067] According to the semantic features, a multi-dimensional semantic vector is constructed, and the task type and urgency features in the multi-dimensional semantic vector are weighted to determine a weighted semantic vector;

[0068] The weighted semantic vector is compared with a preset template library, and a matching template is fused with the semantic features to obtain operation semantic representations.

[0069] It should be noted that the real-time operation sequence includes a sequence of instructions sent by the host, input / output request parameters, and user operation behavior records, wherein the input / output request parameters include request initiation time, target data block ID, and instruction type. The real-time operation sequence is segmented and processed according to single task operations to ensure that each segment contains a complete single task operation. Key words and context information are extracted from each segment, wherein the key words are core action words in each segment that contain task types and urgency levels, such as “fault declaration” and “emergency”. The context information includes the initiating user role, task submission time, data block access frequency, and request initiation time. The key words and context are integrated into semantic features of “key words + context information”, wherein the statistical period of the data block access frequency is set to 1 hour.

[0070] wherein the multi-dimensional semantic vector includes five dimensions of task type, urgency level, user role, data block access frequency, and request initiation time, and the semantic features of each dimension are converted into numerical values to form a vector wherein the conversion into numerical values is as follows: the task type is encoded according to categories, such as 0 for regular query and 1 for fault declaration; the user role is encoded according to permission levels, such as 0.8 for operation and maintenance personnel, 0.5 for ordinary employees, and 0.2 for visitors; the request initiation time is encoded according to load periods; the urgency level is assigned a value according to keyword matching, such as 1.0 for keywords including “emergency fault” and “production line shutdown”, 0.5 for keywords including “regular maintenance”, and 0.1 for no emergency identifier; and the data block access frequency is assigned a value by dividing the actual access frequency by the maximum access frequency. The weighting processing method is to set weights according to the importance of each dimension to task decision-making wherein the importance of each dimension to task decision-making is determined by the correlation between each dimension feature and the actual priority of the task in the past three months, the greater the absolute value of the correlation, the higher the importance and the higher the weight, and the weighted vector is calculated by the following mathematical formula:

[0071] ,

[0072] wherein, is a vector representing the semantic features of the five dimensions, is the weight of each dimension. The preset template library includes standard task vectors, such as an emergency fault declaration template vector wherein the preset template library takes the operation sequence with the frequency greater than or equal to 50 times in the past 3 months as a template candidate scene, takes the average value of the multi-dimensional semantic vector of each candidate scene as a standard task vector, calculates the Euclidean distance of the weighted vector and each template, selects the template with the minimum Euclidean distance as the matching template, combines the attributes of the matching template with the original semantic features, supplements the information not covered by the template, and obtains the operation semantic representation. The operation semantic representation includes task type, urgency, processing priority, associated data block, user role, and request initiation time.

[0073] Exemplarily, in an enterprise operation and maintenance scene, the host end collects a real-time operation sequence: "14:30 operation and maintenance personnel log in the system - select the "device fault" module - input "production line downtime" - submit data read and write request", wherein the target data block of the submitted data read and write request is DB001. First, segment it into 1 segment according to single task operation, extract the keywords "device fault, production line downtime, submit", combine the context "operation and maintenance personnel, 14:30, DB001", obtain the semantic feature "device fault - production line downtime - operation and maintenance personnel - 14:30 - DB001", and then construct a 5-dimensional vector V = [1 (fault declaration), 1 (emergency), 0.8 (operation and maintenance role), 0.9 (DB001 access frequency), 0.9 (14:30 time period)], the original vector is [1, 1, 0.8, 0.9, 0.9], the weighted vector is calculated according to the weight W = [0.3, 0.4, 0.1, 0.15, 0.05] to get [0.3, 0.4, 0.08, 0.135, 0.045], and finally compared with the template library, it is found that the Euclidean distance with the "emergency fault declaration template" is the smallest, the template "priority processing" attribute is fused with the original semantic feature, and the operation semantic representation is obtained.

[0074] In step S2, according to the operation semantic representation, statistical analysis and priority sorting are performed to obtain a hierarchical task queue, including:

[0075] According to the operation semantic representation, the data block access frequency and operation sequence mode are counted to obtain an operation behavior feature set;

[0076] The emergency degree and dependency relationship of the tasks in the operation behavior feature set are compared, if the emergency degree exceeds a preset emergency threshold, the operation parameters are adjusted to determine a task list;

[0077] For the task list, adjustment and scheduling hierarchy are performed to obtain a hierarchical task queue.

[0078] It should be noted that the data block access frequency statistics are counted in a fixed time window, and the fixed time window size is set to 1 hour, and the access frequency is the number of times of accessing the data block in the time window divided by the size of the time window. The operation sequence mode judges the similarity by calculating the Euclidean distance between the multi-dimensional semantic vectors, and when the Euclidean distance between two multi-dimensional semantic vectors is less than a preset distance threshold, the task types represented by the two multi-dimensional semantic vectors are classified into a class. The operation behavior feature set includes the data block access frequency, operation mode category, task execution time, queue waiting time, and emergency degree evaluation value of each task, forming structured feature data. The preset distance threshold is determined based on the classification effect verification of historical operation sequence data. Specifically, from the operation logs in the past 3 months, extract all multi-dimensional semantic vector groups labeled as the same task type, and the task type is divided according to the task type code in the operation semantic representation, such as consistent coding, which is considered as the same type; for each vector group, calculate the Euclidean distance between all vectors in the group, and take the average as the average Euclidean distance of the group. Subsequently, k-fold cross-validation (with a coefficient of 5) is used for effect verification, and the historical vector set is randomly divided into 5 subsets, and 4 subsets are used as the training set and 1 subset is used as the test set; in the training set, the average distance is scaled to generate candidate threshold values with different offsets (such as 10%, 15%, 20%, 25%, 30%), and the threshold values are used to cluster the test set vectors, and the vectors with a Euclidean distance less than the threshold value are classified into the same class, and the accuracy of the clustering result and the true task type label is calculated; finally, select the offset that makes the average accuracy highest and reaches more than 95%, if multiple offsets meet, take the minimum value to ensure the strictness of the threshold value. The final threshold value is the product of the global average of all group average Euclidean distances and the selected offset. Exemplarily, if the global average of the average distance of the historical same class vectors is 0.3, and the verification shows that the 20% offset makes the accuracy reach 96%, then the threshold value is 0.3x1.2=0.36.

[0079] The comparison manner of comparing the urgency and dependency of the tasks in the operation behavior feature set is that the urgency is directly extracted from the urgency evaluation value in the operation behavior feature set, the dependency is judged by the tasks and their associated data blocks, the input data block ID and the output data block ID of each task are extracted according to the associated data block information in the operation semantic representation, if the output data block ID of task one is completely consistent with the output data block ID of task two, it is judged that task two depends on task one, and task one should be executed first. The preset emergency threshold is determined by historical data statistics, and the minimum value of the urgency of high-priority tasks in the past three months is taken, such as 0.6, wherein the high-priority task is defined as a task with an urgency evaluation value greater than or equal to 0.6 and a data block access frequency greater than or equal to 5 times per hour in the operation semantic representation. If the task urgency exceeds the preset emergency threshold, the resource allocation weight parameter of the task is adjusted, and the step is 0.1, until the task that meets the condition of urgency greater than or equal to the preset emergency threshold is included in the task list after adjustment, or the maximum adjustment times is reached, then it is forced to be included in the task list, wherein the maximum adjustment times is set to 4 times.

[0080] For the task list, adjustment and scheduling layering are performed, wherein the adjustment and layering manner is that the task list is sorted in descending order of resource allocation weight first, and then combined with operation execution frequency to divide levels, the frequency statistical period is 1 hour, that is, tasks with execution frequency greater than or equal to 5 times per hour are set as high-frequency tasks, the rest are low-frequency tasks, the weight greater than or equal to 0.7 is high weight, the weight less than 0.7 and greater than 0.4 is medium weight, and the weight less than 0.4 is low weight, then the tasks with high frequency and high weight are set as upper layer queue, the tasks with low frequency and medium weight are set as middle layer queue, and the tasks with low frequency and low weight are set as lower layer queue. The formation of the layered task queue is to sort the tasks in the same level according to the task dependency relationship, and to schedule different levels according to the rule of upper layer priority, and finally to form a multi-level and ordered layered task queue.

[0081] Exemplarily, in an IT operation and maintenance system, the operation semantic representation corresponds to three tasks, task 1, task 2 and task 3. The urgency of task 1 is 0.8, the data block access frequency is 10 times / hour, the execution time is 4 hours, the urgency of task 2 is 0.3, the access frequency is 2 times / hour, and it depends on task 1, the urgency of task 3 is 0.6, the access frequency is 4 times / hour, and the execution time is 2 hours. The preset emergency threshold is 0.6, the urgency of tasks 1 and 3 exceeds 0.6, the resource allocation weight of the tasks is gradually adjusted from 0.5 to 0.8 with a step of 0.1, the task list is sorted in descending order of weight as [task 1, task 3, task 2], then the high frequency and high weight are divided into the upper layer, that is, task 1, the low frequency and medium weight are divided into the middle layer, that is, task 3, the low frequency and low weight are divided into the lower layer, that is, task 2, and the layered task queue is formed.

[0082] In step S3, the layered task queue is dynamically adjusted in storage resources to obtain a resource distribution structure, including:

[0083] Based on the preset proportion threshold, the layered task queue is divided into tasks, the data block access frequency is analyzed for relevance, and a data block set is determined.

[0084] According to the data block set, the buffer allocation is optimized and hierarchical management is performed to obtain a resource distribution structure.

[0085] It should be noted that the preset proportion threshold is determined based on the task processing efficiency data in the historical high load scenario of the solid state storage. The task processing data in the high load period of the system in the past 3 months is counted, wherein the high load period is from 10:00 to 11:00 every day. The "resource utilization rate-response delay" curve under different high priority task proportions is calculated, and the minimum proportion of resource utilization rate ≥ 85% and response delay ≤ 50ms in the curve is taken as the preset proportion threshold. The high priority task is defined as a task with an emergency degree evaluation value greater than or equal to 0.6 and a data block access frequency greater than or equal to 5 times per hour. For example, it is found that when the proportion of high priority tasks is 20%, the resource utilization rate is 88% and the response delay is 45ms. The delay increases sharply when the proportion exceeds 20%, so the preset proportion threshold is set to 20%.

[0086] The division of the layered task queue is based on whether the task belongs to a high priority task. When dividing, the total number of high priority tasks in the layered task queue is first counted, and then the proportion of the total number of high priority tasks in the total number of tasks in the queue is calculated, that is, the proportion of high priority tasks is the number of high priority tasks divided by the total number of tasks in the queue. If the proportion of high priority tasks is greater than the preset proportion threshold, the high priority tasks exceeding the threshold are temporarily classified as high priority tasks to be adjusted, the part not exceeding the threshold is retained as core high priority tasks, and the rest are regular tasks, thereby completing the task division.

[0087] The relevance analysis of the data block access frequency is realized by calculating the co-occurrence frequency of "task-data block", which is calculated by the following mathematical formula:

[0088] ,

[0089] Wherein, K represents the number of simultaneous access to task T and data block D within 1 hour, M represents the total number of access to task T within 1 hour. The closer C is to 1, the stronger the association between data block D and task T. First, extract all data block IDs associated with the divided core high priority task, then calculate the co-occurrence frequency of each data block and the core high priority task, if the co-occurrence frequency is greater than or equal to the preset association threshold 0.8, the data block is included in the high-frequency associated data block subset, and the data block belonging to the high-frequency mode is added to the operation sequence mode classification at the same time. Finally, the data block set is obtained by merging, wherein the preset association threshold is determined based on historical data, the operation log of the core high priority task in the past 3 months is extracted, the co-occurrence frequency of each core task and its associated data block is calculated, the response delay increment of the core task when the data block is not loaded in time is counted in different co-occurrence frequency intervals, and the co-occurrence frequency corresponding to the delay increment less than or equal to 10% is taken as the preset association threshold.

[0090] It is worth noting that the buffer optimization allocation mode based on the data block set is to first count the average access time of each data block in the data block set, that is, the total access time divided by the number of accesses, and then allocate resources according to the principle that the longer the average access time, the larger the buffer capacity. The total capacity of the buffer is B, and the capacity allocated to a single data block is represented by the following mathematical formula:

[0091] ,

[0092] Wherein, is the allocated capacity of the i-th data block, B is the total capacity of the buffer, such as 1000MB, is the average access time of the i-th data block, n is the total number of data blocks, is the actual size of the i-th data block. The hierarchical management mode based on the data block set is to divide the storage hierarchy according to the access frequency of the data block. The data block with an access frequency greater than or equal to 10 times / hour is classified into the cache layer, i.e. the read / write speed is ≥100MB / s. The data block with an access frequency greater than or equal to 5 times / hour and less than 10 times / hour is classified into the ordinary storage layer, i.e. the read / write speed is ≥50MB / s. The data block with an access frequency less than 5 times / hour is classified into the low-speed storage layer, i.e. the read / write speed is ≥20MB / s. At the same time, the storage location, allocated capacity and access authority of each level data block are recorded to ensure that the high-frequency associated data block is in the high-speed level. The resource allocation result and the hierarchical management data are integrated to form a structured resource distribution structure, including the "level attribution-allocated capacity-storage location" mapping table of each data block, the "total capacity-used capacity-remaining capacity" statistics of each storage level, and the resource scheduling rule. Finally, the resource distribution structure is output as a readable structured file, wherein the resource scheduling rule is that the cache layer data block responds to the core high priority task first.

[0093] Exemplarily, the layered task queue of an IT operation and maintenance system contains 100 tasks, among which there are 25 high-priority tasks with an urgency level ≥ 0.6 and an access frequency ≥ 5 times / hour, accounting for 25%, which exceeds the preset proportion threshold of 20%, so 5 high-priority tasks are classified as to-be-adjusted high-priority tasks, and 20 are classified as core high-priority tasks. A 10% buffer resource is reserved for to-be-adjusted high-priority tasks, and the resource utilization rate of core high-priority tasks is monitored in real time. When it is detected that the resource utilization rate of core high-priority tasks is lower than 80%, the first two to-be-adjusted high-priority tasks are activated and classified into the core high-priority task queue in descending order of urgency level. The data blocks D1 and D2 associated with the core high-priority tasks are extracted, wherein D1 has an access frequency of 12 times / hour and an actual size of 100 MB, and D2 has an access frequency of 8 times / hour and an actual size of 50 MB. The co-occurrence frequency of D1 and D2 with the core high-priority tasks is calculated to be 0.9, and D1 and D2 are included in the high-frequency associated data block subset. At the same time, the data block D3 associated with the "emergency fault declaration" mode in the operation sequence mode classification is added, and D3 has an access frequency of 7 times / hour and an actual size of 80 MB. Finally, the data block set is determined to be {D1, D2, D3}. The total capacity of the buffer is 1000 MB, the average access time of D1 is 0.8 s, the average access time of D2 is 0.2 s, and the average access time of D3 is 0.5 s. According to the formula, D1 is allocated to be 615 MB, D2 is allocated to be 77 MB, and D3 is allocated to be 308 MB. D1 is classified into the cache layer because of the access frequency of 12 times / hour, and D2 and D3 are classified into the ordinary storage layer. After integration, the resource distribution structure is formed, and the hierarchy, capacity and storage location of each data block are determined.

[0094] In step S4, according to the resource distribution structure, a space recycling operation is performed and the resource utilization rate is monitored to determine the potential resource bottleneck position, and a storage space mapping is obtained, including:

[0095] Based on the resource distribution structure, the task-associated data blocks that meet the preset judgment condition are extracted and prioritized to obtain a to-be-processed data block set;

[0096] The storage location of the data blocks in the to-be-processed data block set is reordered to determine the data block distribution;

[0097] If the data block distribution does not meet the preset distribution standard, the storage resource occupation proportion corresponding to the resource distribution structure is analyzed to determine the potential resource bottleneck position;

[0098] According to the potential resource bottleneck position, the data block storage location is dynamically adjusted to obtain a storage space mapping.

[0099] It is worth noting that the data block extraction is first screened from the resource distribution structure associated with the core high priority task, and then filtered according to the preset judgment condition, and the data block meeting the condition is reserved, wherein the preset judgment condition is set based on the support degree of the data block to the core task, and the preset judgment condition includes two quantitative indicators, one is that the co-occurrence frequency of the data block and the core high priority task is greater than or equal to 0.8, and the other is that the storage time of the data block is greater than or equal to 24 hours. The priority sorting is based on the priority score, and the priority score is calculated by the following mathematical formula:

[0100] ,

[0101] Wherein, C is the co-occurrence frequency, is the normalized access frequency, that is, the actual access frequency of the data block divided by the maximum access frequency of the core high priority task associated data block in the past 3 months, which is 20 times per hour, set to 0.6, set to 0.4, because the co-occurrence frequency has greater impact on the core task. According to the descending order of priority score, the top 80% of the core high priority task associated data block is taken from the total number of data blocks to form the to-be-processed data block set.

[0102] It should be noted that the way of reordering the storage location of the data block in the to-be-processed data block set is to read the "current storage address", "storage level" and "adjacent data block access correlation" of each data block in the resource distribution structure, wherein the "adjacent data block access correlation" is the probability that two adjacent data blocks are continuously accessed by the same task. According to the principle that "the physical addresses of data blocks with high correlation in the same level are adjacent", the access correlation degree of any two data blocks is calculated, that is, the number of times that the same task continuously accesses two data blocks is divided by the sum of the total access times of the two data blocks, and the data blocks with access correlation degree greater than or equal to 0.7 are divided into a group. Within the same group, the addresses are sorted from small to large, and different groups are allocated physical address intervals from high to low according to the access frequency. After sorting, record the "new storage address", "belonging storage level", "belonging correlation group" and "address interval with adjacent data block" of each data block to form a structured "data block-location mapping table", that is, the data block distribution.

[0103] The preset distribution standard is determined by historical data statistics and includes two core indexes: one is that the average address interval of the data blocks in the same level is less than or equal to 10 KB, and the other is that the data block distribution uniformity of each storage level is greater than or equal to 0.8, wherein the uniformity is the number of data blocks in the level divided by the maximum number of data blocks that can be accommodated in the level, and the maximum number of data blocks that can be accommodated in the level of the solid-state storage refers to the maximum number of data blocks that can be simultaneously stored in a certain storage level of the solid-state storage under the hardware specification limit. For example, taking the cache layer as an example, the total capacity is 1000 MB, and the standard size of a single data block is 50 MB, so the maximum number of data blocks that can be accommodated in the cache layer is 1000 MB / 50 MB = 20. The average size of a single data block is set by statistically analyzing the data blocks associated with the core high-priority tasks in the past three months. If the data block distribution does not meet the preset distribution standard, the storage resource occupation proportion corresponding to the resource distribution structure is analyzed, that is, the total capacity, used capacity and data block quantity of each storage level are extracted from the resource distribution structure, the occupation proportion is calculated, and the occupation proportion is the percentage of the used capacity divided by the total capacity. If the occupation proportion of a certain level is greater than or equal to 90%, or the data block distribution uniformity of the level is less than 0.8, that is, the preset distribution standard is not met, it is determined that the level is a potential resource bottleneck position.

[0104] The dynamic adjustment mode based on the potential resource bottleneck position is that, for the potential bottleneck position, first, the low-association data blocks with an access frequency less than 3 times / hour in the level are selected and migrated to a lower storage level to release the capacity of the bottleneck level, and then the high-association data blocks with an access frequency greater than or equal to 10 times / hour in the to-be-processed data block set are migrated to the idle address of the bottleneck level to ensure that the occupation proportion of the bottleneck level is reduced to below 85% and the uniformity is improved to 0.8 or above. The "adjusted data block-new storage address mapping table", "capacity occupation data of each storage level" and "data block access permission configuration" are integrated to form a structured storage space mapping.

[0105] Exemplarily, in a certain system resource distribution structure, 20 core high-priority task associated data blocks are screened according to a preset determination condition, and it is obtained through statistics that the historical maximum access frequency is 20 times per hour. According to the priority score formula, the D1 data block information is C=0.9, f=12 times / hour, the normalized access frequency is 0.6, S=0.9*0.6+0.6*0.4=0.78, ranked first; the D2 data block information is C=0.85, f=10 times / hour, the normalized access frequency is 0.5, S=0.85*0.6+0.5*0.4=0.71, ranked second; the D3 data block information is C=0.82, f=8 times / hour, the normalized access frequency is 0.4, S=0.82*0.6+0.4*0.4=0.652, ranked third, forming a set of data blocks to be processed. The data blocks in the set are reordered, and the data blocks D1, D2 and D3 with an access correlation greater than or equal to 0.7 are divided into a group, the addresses are sorted in the order of 100KB-130KB, the data block distribution is obtained, it is found through detection that the occupation ratio of the cache layer is 92%≥90%, the uniformity is 0.7<0.8, and it is determined that it is a bottleneck position. The two low-correlation data blocks in this level are migrated to the general storage layer, and D1-D3 are migrated in. After adjustment, the occupation ratio of the cache layer is 82%, and the uniformity is 0.85. The data block-address mapping and level capacity data are integrated to obtain a storage space mapping. The storage space mapping includes core information such as "data block ID-physical storage location", "bandwidth of each storage level", and "data block access frequency". The physical storage location includes the level and address of the data block.

[0106] In step S5, according to the storage space mapping, the cache access path is updated, and the data access channel is determined, including:

[0107] According to the storage space mapping, path planning is performed to determine a path allocation scheme;

[0108] The path allocation scheme is subjected to time smoothing processing to obtain a smoothed fluctuation range, and a delay tolerance is determined;

[0109] If the delay tolerance does not reach a preset tolerance threshold, the cache access path is adjusted, and the data access channel is determined.

[0110] It should be noted that the path planning aims to minimize access delay, and A *The algorithm implementation is that, taking the "data block request initiation node" as the starting point and the "data block physical storage address" as the end point, a path search space is constructed, each node in the space represents an address segment of a storage level, exemplarily, the storage space mapping contains 3 storage levels, which are cache layer, normal storage layer and low-speed storage layer, wherein the address range of the cache layer is 0x0000-0x0FFF, the address range of the normal storage layer is 0x1000-0xFFFF, and the address range of the low-speed storage layer is 0x10000-0xFFFFF, the address range of node one is 0x0000-0x0063 representing the cache layer, the address range of node two is 0x1000-0x11F3 representing the normal storage layer, and the low-speed storage layer is divided into 256 nodes at an interval of 1MB, taking the request to access the cache layer node one as an example, the starting point of the path search space is "host request initiation node" and the end point is "cache layer node one", the candidate path needs to traverse the possible paths such as "host→cache layer interface→node one" and "host→normal storage layer interface→cache layer node one", and each address segment of the storage level is in the form of an independent node participating in path evaluation. The next node is selected from small to large according to the evaluation value until the end point is searched, generating 1-3 candidate access paths, wherein the evaluation value is represented by the following mathematical formula:

[0111] ,

[0112] wherein, is an evaluation function, is the actual access delay from the starting point to the current node n, which is calculated based on the bandwidth of the storage level, that is, the data quantity is divided by the total bandwidth of a certain level, for example, when the data quantity is 10MB and the bandwidth of the cache layer is 100MB / s, which is equal to 10MB divided by 100MB / s, equal to 0.1s. is the estimated delay from the current node n to the end point, which uses Euclidean distance as a heuristic function, specifically, the physical address distance between node n and the end point is divided by the maximum bandwidth of the storage level, for example, when the bandwidth of the cache layer is 100MB / s, it is ensured that is not greater than the actual minimum delay. For example, if the address distance between node n and the end point is 1KB and the maximum bandwidth is 100MB / s, The path stability score of the candidate access path is calculated, the path stability score is obtained by dividing 1 by 1 plus the sum of the standard deviation of the delay fluctuation within 1 hour of the path, and the path stability score is sorted in descending order, high-frequency data blocks with access frequency greater than or equal to 10 times / hour are preferentially assigned to paths with path stability scores greater than or equal to 0.8, medium-frequency data blocks with access frequency less than 10 times and greater than or equal to 5 times are assigned to paths with stability greater than or equal to 0.7, and low-frequency data blocks with access frequency less than 5 times / hour are assigned to paths with path stability greater than or equal to 0.6, to form a path allocation scheme containing "data block ID-candidate path-assignment priority".

[0113] It is worth noting that the time smoothing processing of the path allocation scheme adopts a sliding window average method, the length of the time window is set to 10 seconds, real-time access delay data of each path in the path allocation scheme is collected, the average value of the delay within the window is calculated, and the standard deviation is calculated. The delay tolerance is defined as a preset tolerance threshold, which is based on the historical core high priority task delay distribution. From the data in the past 3 months, the 95% quantile delay value (such as 25ms) is taken as the reference, and then a coefficient is multiplied according to the safety margin requirement, and the coefficient is greater than or equal to 1. Through load stress test, under peak load, the coefficient value is gradually increased until the 95% task delay meets the preset service level agreement (SLA) requirement, and the minimum coefficient value that meets the condition is taken as the final coefficient; for example, when the coefficient is equal to 1.1, the threshold is 27.5ms. The average delay in the sliding window is only used to evaluate the path fluctuation, and is not used to define the tolerance. If the delay tolerance is less than the preset tolerance threshold, it means that the current path delay is too low, and part of the bandwidth resource can be released, and if the delay tolerance is greater than the preset tolerance threshold, the adjustment is made according to the rules.

[0114] The rules are to preferentially migrate low-frequency data blocks with access frequency less than 3 times per hour on the path to other low-load paths; the standard for judging low frequency is based on analyzing the access frequency of all data blocks in the past 3 months, when the access frequency is less than 3 times / hour, the average resource utilization of the data block is less than 40%, which belongs to inefficient occupation; the specific operation of releasing bandwidth resources is to preferentially migrate the data block with the highest weight on the path to other paths, or to reduce the scheduling priority weight of the path, such as reducing the weight from 40% to 30%, until the delay tolerance approaches the preset tolerance threshold.

[0115] If the operation is still not satisfied, the scheduling priority of the current path is adjusted, such as by the weight value in the weighted fair queue, to allocate more time slice resources preferentially. Exemplarily, the path priority weight is increased from 30% to 40%, which actually means that the proportion of processing time requested by the path is increased, and the delay tolerance is reduced from 28 ms to 25 ms. By integrating the "path that meets the delay requirement after adjustment", the "storage level interface corresponding to the path" and the "data block access authority", the paths with "delay tolerance ≤ preset tolerance threshold and stability score ≥ 0.7" are screened out, each effective path corresponds to an independent data access channel, and finally a mapping table of "data block ID-data access channel-channel bandwidth" is formed to determine the final data access channel.

[0116] In step S6, multi-thread conflicts in the data access channel are detected and adjusted to obtain a response time distribution, including:

[0117] If there is a multi-thread conflict in the data access channel, the detailed position of the conflict node is obtained, and the priority of the storage resource corresponding to the resource distribution structure is adjusted according to the detailed position of the conflict node to obtain a resource distribution scheme;

[0118] If the resource distribution scheme cannot meet the demand of concurrent requests, the priority of the storage resource is adjusted to obtain a request processing queue;

[0119] The request processing queue is subjected to time smoothing processing to obtain a response time distribution.

[0120] It should be noted that the judgment method of multi-thread conflict is to determine by monitoring the thread waiting time and resource competition times of data access channel in real time, to count the average time of thread waiting for resources in each channel, and the number of times of thread competition for the same resource per unit time, if the average time of thread waiting for resources is greater than the preset waiting threshold and the number of times of thread competition for the same resource is greater than the preset competition threshold, it is determined that there is multi-thread conflict, wherein the preset waiting threshold is determined based on the maximum waiting time of historical non-conflict period, such as the average time of thread waiting for resources in non-conflict time is 80ms, and the maximum is set to 100ms with a 25% increase, and the preset competition threshold is twice the average number of times of historical non-conflict period, such as 5 times per second. The acquisition method of the detailed position of the conflict node is to extract the physical storage address range and storage level interface corresponding to the data access channel from the storage space mapping, and to locate the conflict node combined with the thread monitoring log, the log records the target data block ID, request time and resource occupation state of the thread request, and the threads whose average time of waiting for resources is greater than 100ms and the number of times of thread competition for the same resource is greater than 5 times per second are screened out, and the data block storage address and storage level interface requested by the threads are the detailed position of the conflict node. The storage resource allocation priority adjustment rule is based on the task semantic characteristics and the resource state of the conflict node, that is, to extract the urgency evaluation value and data block access frequency of the conflict node associated task, to calculate the priority score, and the priority score is calculated by the following mathematical formula:

[0121] ,

[0122] wherein, is the urgency evaluation value, is the normalized access frequency, that is, the actual access frequency of the data block divided by the maximum access frequency of the core high priority task associated data block in the last 3 months, which is counted as 20 times per hour, is the weight set to 0.6, is the weight set to 0.4. The storage resource allocation priority is adjusted in descending order of priority score, that is, the task with a priority score greater than or equal to 0.8 is given priority to get the conflict node resource, and the remaining tasks are temporarily adjusted to other non-conflict nodes, and the "adjusted priority order", "conflict node resource allocation ratio" and "non-conflict node task migration scheme" are integrated to form the resource distribution scheme.

[0123] The manner of judging whether the resource distribution scheme meets the demand of concurrent requests is to calculate a concurrent request processing rate, which is the number of successfully processed concurrent requests in a window divided by the total number of concurrent requests in the window, wherein the window is set to 10 seconds. If the concurrent request processing rate is less than a preset processing rate threshold, or the average length of time that threads wait for resources is greater than a secondary waiting threshold, it is judged that the resource distribution scheme cannot meet the demand. The preset processing rate threshold is determined based on the historical lowest processing success rate and is set to 90%, and the secondary waiting threshold is 1.5 times the initial preset waiting threshold and is set to 150 ms. When storing the resource allocation priority adjustment, all concurrent requests are first sorted in descending order of priority scores, and then a waiting time compensation factor is introduced to modify the priority scores. The waiting time compensation factor is equal to 1 plus the length of time that the thread has been waiting divided by the secondary waiting threshold. The modified priority score is the priority score multiplied by the waiting time compensation factor. The requests are arranged in descending order of the modified priority scores, and the requests with the same score are sorted according to the request time in chronological order to form a request processing queue.

[0124] It is worth noting that the time smoothing processing of the request processing queue is to process the response time of each request in the queue by using the exponential weighted moving average method, which is calculated by the following mathematical formula:

[0125]

[0126] wherein, represents the smoothed response time of the i-th request, represents the actual response time of the i-th request, is a smoothing coefficient, which is set to 0.3, is set to the actual response time of the first request in the queue. The smoothed response times of all requests are counted and presented in the form of response time interval- corresponding request quantity proportion to form a response time distribution.

[0127] ​Exemplarily, a certain data access channel monitors threads T1 and T2, wherein T1 requests data block D1, the urgency evaluation value is 0.9, the data block access frequency is 12 times / hour, T2 requests data block D2, the urgency evaluation value is 0.5, and the data block access frequency is 3 times / hour. The average duration of thread waiting for resources is 120 ms, and the number of times of thread competition for the same resource is 6 times / s. It is determined that there is a conflict, and the conflict node is located at cache layer 0x001-0x005. The priority score is calculated, T1 gets 0.78, and T2 gets 0.36. The resource is preferentially allocated to T1, and a resource distribution scheme is generated. The concurrent request processing rate within 10 seconds is 85%<90%, a waiting time compensation factor is introduced, T1 has waited for 120 ms, the waiting time compensation factor is 1.8, T2 has waited for 130 ms, the waiting time compensation factor is 1.87, the score of T1 after correction is 1.404, and the score of T2 is 0.672. The request processing queue is formed in descending order of score. The response time of the queue is smoothed by an exponential weighted moving average method, and a response time distribution is generated.

[0128] In step S7, according to the response time distribution, the resource utilization rate is tracked, the semantic parameters are dynamically updated and the adjustment range is limited, and a data processing instruction is obtained, including:

[0129] According to the response time distribution, the resource utilization rate is tracked, the specific period of resource occupation peak is obtained, the update range of the semantic parameters is processed in stages for the specific period, if the update range exceeds a preset range threshold, the parameter adjustment frequency is limited, and a parameter configuration scheme is obtained;

[0130] According to the parameter configuration scheme, the priority of the semantic parameters is sorted, and the parameter application order is determined;

[0131] According to the parameter application order, the processing demand of the real-time operation sequence is loaded step by step, and a data processing instruction is obtained.

[0132] It should be noted that the specific period of tracking resource utilization and obtaining resource occupation peak value first sets the time granularity as 5 minutes, and the average response time and resource utilization in each granularity are counted. The resource utilization is the actual occupied resource amount divided by the total resource amount, wherein the actual occupied resource amount refers to the actual occupied amount of hardware resources by the current task within 5 minutes, and the total resource amount refers to the maximum hardware carrying capacity of the solid-state memory core hardware resources, which is a fixed value determined based on the hardware specifications when the system is shipped. If the average response time of a certain period is greater than the preset response threshold and the resource utilization is greater than the preset utilization threshold, it is marked as a high load period, wherein the preset response threshold is determined based on 1.5 times the average response time of the historical normal response period, and is set to 120 ms. The preset utilization threshold is determined based on 85% of the upper limit of the historical resource hardware carrying capacity, such as 100 MB / s of storage bandwidth upper limit, and is set to 85%. Further refine the time granularity in the high load period to 1 minute, and take the period corresponding to the maximum resource utilization as the resource occupation peak period. The semantic parameters include the emergency degree weighting coefficient and the access frequency weighting coefficient, which are obtained in step S1 of constructing the semantic vector. The update range of the semantic parameters is processed based on the resource utilization of the resource occupation peak period, and when the resource utilization of the resource occupation peak period is less than or equal to 88%, it is divided into level one, and the update range is less than or equal to 10% by increasing the weighting coefficient in the semantic parameter; when the resource utilization is greater than 88% and less than or equal to 92%, it is divided into level two, and the update range is less than or equal to 15% by further increasing the weighting coefficient in the semantic parameter; when the resource utilization is greater than 92%, it is divided into level three, and the update range is less than or equal to 20% by further increasing the weighting coefficient in the semantic parameter. The preset amplitude threshold is determined based on the influence of semantic parameter adjustment on the system, and the maximum amplitude corresponding to "parameter adjustment response time fluctuation ≤ 10%" is taken as 25%. If the update range of the semantic parameter exceeds 25%, the adjustment frequency is limited from real-time adjustment to fixed period adjustment, and the fixed period is calculated by the following mathematical formula:

[0133] ,

[0134] wherein, is the basic period, which is determined based on the historical stable adjustment period and is set to 5 minutes, F is the actual amplitude, is the preset amplitude threshold. The generation of the parameter configuration scheme integrates the "graded update range", "limited adjustment frequency" and "specific adjustment value of each semantic parameter", thereby forming a structured scheme.

[0135] wherein, the semantic parameter priority sorting is based on the contribution of the parameter to the optimization of resource utilization, and the contribution is calculated by the change rate of resource utilization before and after parameter adjustment, which is represented by the following mathematical formula:

[0136] ,

[0137] wherein, is the resource utilization after parameter adjustment, is the resource utilization before parameter adjustment. The contribution values of each semantic parameter in the parameter configuration scheme are extracted, and the contribution values are sorted in descending order. The priority of the parameter with high contribution value is high, and then the priority order is obtained. After the priority order is determined, the application order is determined according to the dependency relationship between the parameters. If the adjustment of parameter one requires the stability of parameter two as a prerequisite, then parameter two is applied first. If there is no dependency relationship, then it is directly applied according to the priority from high to low. Exemplarily, if the parameter configuration scheme contains two semantic parameters, which are the emergency degree weighting coefficient and the access frequency weighting coefficient, the original value of the emergency degree weighting coefficient is 0.6, and the original value of the access frequency weighting coefficient is 0.4. First, the contribution degree is calculated to determine the priority. The resource utilization before adjustment is 92%. The emergency degree weighting coefficient is adjusted to 0.7, and the resource utilization after adjustment is 82%. The contribution degree is 6.52%. The access frequency weighting coefficient is adjusted to 0.5, and the resource utilization after adjustment is 89%. The contribution degree is 3.26%. According to the descending order of the contribution degree, the priority of the emergency degree weighting coefficient is higher than that of the access frequency weighting coefficient. Then the dependency relationship is determined. In the semantic vector construction logic, the adjustment of the access frequency weighting coefficient requires the value of the emergency degree weighting coefficient to be stable as a prerequisite. Therefore, the application order is to apply the emergency degree weighting coefficient with high priority and as the basis of the dependency first. Then the access frequency weighting coefficient is applied after it is stable.

[0138] It is worth noting that the distribution loading mode of the real-time operation sequence is to divide the operation sequence loading batches based on the parameter application order, which are the first batch, the second batch and the third batch. The first batch loads the operation sequence that only depends on the high-priority parameter. The high-priority parameter is the parameter whose priority is the first 40% of the total sequence parameters. 40% is determined based on historical data statistics. That is, in the past three months, the adjustment of the top 40% high-contribution-degree parameters can solve more than 75% of the resource load problem. Therefore, it is set to 40%. The total sequence parameters include the emergency degree weighting coefficient, the access frequency weighting coefficient, the user role weight, the request time weight, etc. The third batch loads the operation sequence that depends on the low-priority parameter. The low-priority parameter is the parameter whose priority is the last 25% of the total sequence parameters. The remaining operation sequence is loaded in the second batch. The generation of the data processing instruction needs to integrate the "operation sequence ID loaded in each batch", "corresponding parameter application state" and "resource monitoring data after loading" to form the data processing instruction.

[0139] In summary, the application provides a solid-state memory data processing method, which solves the problem of prominent system overall performance bottleneck under complex dynamic workloads in the prior art through semantic analysis, dynamic resource adjustment and conflict coordination.

[0140] Referring to Figure 2 The second embodiment of the application provides a solid-state memory data processing system, comprising:

[0141] An operation semantic capturing module collects real-time operation sequences, performs semantic analysis on the real-time operation sequences, and obtains operation semantic representations;

[0142] A hierarchical task queue generating module performs statistical analysis and priority sorting according to the operation semantic representations, and obtains a hierarchical task queue;

[0143] A resource distribution adjusting module performs dynamic adjustment of storage resources on the hierarchical task queue, and obtains a resource distribution structure;

[0144] A storage space mapping optimizing module performs space recycling operations and monitors resource utilization according to the resource distribution structure, determines potential resource bottleneck positions, and obtains a storage space mapping;

[0145] A data access channel determining module updates cache access paths and determines data access channels according to the storage space mapping;

[0146] A concurrent conflict coordinating module detects and adjusts multithreading conflicts in the data access channels, and obtains response time distributions;

[0147] A flow refining module traces resource utilization according to the response time distributions, dynamically updates semantic parameters and limits adjustment amplitudes, and obtains data processing instructions.

[0148] It should be noted that the solid-state memory data processing system provided by the embodiments of the application is used to execute all flow steps of the solid-state memory data processing method provided by the above-mentioned embodiments, and the working principles and beneficial effects of the two are one-to-one corresponding, and thus will not be described again.

[0149] The embodiments of the application also provide an electronic device. The electronic device comprises a processor, a memory, and a computer program, for example, a solid-state memory data processing program, stored in the memory and executable on the processor. The processor implements the steps in the above-mentioned various solid-state memory data processing method embodiments when executing the computer program, for example Figure 1 The step S1 shown. Alternatively, the processor implements the functions of the modules / units in the above-mentioned various device embodiments when executing the computer program, for example, a data access channel determining module.

[0150] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0151] The electronic device can be a computing device such as a desktop computer, a notebook computer, a palm computer, and a smart tablet. The electronic device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above components are only examples of the electronic device and do not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the above, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, etc.

[0152] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor is the control center of the electronic device, which connects all parts of the electronic device through various interfaces and lines.

[0153] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the electronic device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0154] The modules / units integrated in the electronic device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can realize the steps of the above-mentioned various method embodiments when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0155] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate units can or can not be physically separate, and the units displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0156] The above specific embodiments further illustrate the purpose, technical scheme and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of data processing for a solid state memory, characterized by, The method comprises the following steps: collecting real-time operation sequences, performing semantic analysis on the real-time operation sequences to obtain operation semantic representations; performing statistical analysis and priority sorting according to the operation semantic representations to obtain a hierarchical task queue; performing dynamic adjustment of storage resources on the hierarchical task queue to obtain a resource distribution structure; performing space recycling operations and monitoring resource utilization according to the resource distribution structure to determine potential resource bottleneck positions and obtain a storage space mapping; updating cache access paths according to the storage space mapping to determine data access channels; detecting multi-thread conflicts in the data access channels and adjusting them to obtain a response time distribution; tracking resource utilization according to the response time distribution, dynamically updating semantic parameters and limiting the adjustment range to obtain data processing instructions; wherein the collecting real-time operation sequences, performing semantic analysis on the real-time operation sequences to obtain operation semantic representations comprises: collecting real-time operation sequences, performing segmentation processing on the real-time operation sequences, extracting keywords and context information, and obtaining semantic features; constructing a multi-dimensional semantic vector according to the semantic features, weighting the task type and urgency features in the multi-dimensional semantic vector to determine a weighted semantic vector; comparing the weighted semantic vector with a preset template library, fusing the matching template and the semantic features to obtain the operation semantic representations; wherein the performing space recycling operations and monitoring resource utilization according to the resource distribution structure to determine potential resource bottleneck positions and obtain a storage space mapping comprises: based on the resource distribution structure, extracting task-associated data blocks that meet a preset judgment condition and performing priority sorting to obtain a set of to-be-processed data blocks; reordering the data block storage locations in the set of to-be-processed data blocks to determine data block distribution; if the data block distribution does not meet a preset distribution standard, analyzing the storage resource occupation proportion corresponding to the resource distribution structure to determine potential resource bottleneck positions; performing dynamic adjustment of the data block storage locations according to the potential resource bottleneck positions to obtain a storage space mapping; wherein the detecting multi-thread conflicts in the data access channels and adjusting them to obtain a response time distribution comprises: if there are multi-thread conflicts in the data access channels, obtaining the detailed positions of the conflict nodes, adjusting the storage resource allocation priorities corresponding to the resource distribution structure according to the detailed positions of the conflict nodes to obtain a resource distribution scheme; if the resource distribution scheme cannot meet the needs of concurrent requests, adjusting the storage resource allocation priorities to obtain a request processing queue; performing time smoothing processing on the request processing queue to obtain a response time distribution.

2. The solid state memory data processing method of claim 1, wherein, The performing statistical analysis and priority sorting according to the operation semantic representations to obtain a hierarchical task queue comprises: performing statistics on data block access frequency and operation sequence patterns according to the operation semantic representations to obtain an operation behavior feature set; The operation behavior feature set is compared with the urgency and dependency of the task, if the urgency exceeds a preset urgency threshold, the operation parameter is adjusted, and a task list is determined; The task list is adjusted and scheduled in layers to obtain a layered task queue.

3. The solid state memory data processing method of claim 2, wherein, The layered task queue is dynamically adjusted in storage resources to obtain a resource distribution structure, including: Based on a preset proportion threshold, the layered task queue is divided into tasks, the data block access frequency is analyzed for correlation, and a data block set is determined; According to the data block set, the buffer allocation is optimized and managed in layers to obtain the resource distribution structure.

4. The solid state memory data processing method of claim 1, wherein, According to the storage space mapping, the cache access path is updated, and the data access channel is determined, including: According to the storage space mapping, the path planning is performed to determine the path allocation scheme; The path allocation scheme is time-smoothed to obtain a smoothed fluctuation range and determine a delay tolerance; If the delay tolerance does not reach a preset tolerance threshold, the cache access path is adjusted to determine the data access channel.

5. The solid state memory data processing method of claim 1, wherein, According to the response time distribution, the resource utilization rate is tracked, the semantic parameters are dynamically updated and the adjustment amplitude is limited, and a data processing instruction is obtained, including: According to the response time distribution, the resource utilization rate is tracked, the specific period of the resource occupation peak is obtained, the update amplitude of the semantic parameters is processed in stages for the specific period, if the update amplitude exceeds a preset amplitude threshold, the parameter adjustment frequency is limited, and a parameter configuration scheme is obtained; According to the parameter configuration scheme, the priority of the semantic parameters is sorted to determine the parameter application order; According to the parameter application order, the processing requirements of the real-time operation sequence are loaded step by step to obtain the data processing instruction.

6. A solid state memory data processing system, characterized by, A solid state memory data processing method for implementing any one of claims 1 to 5, comprising: An operation semantic capture module collects a real-time operation sequence, performs semantic analysis on the real-time operation sequence, and obtains an operation semantic representation; A layered task queue generation module performs statistical analysis and priority sorting based on the operation semantic representation to obtain a layered task queue; A resource distribution adjustment module dynamically adjusts the layered task queue in storage resources to obtain a resource distribution structure; A storage space mapping optimization module performs space recycling operations and monitors resource utilization based on the resource distribution structure to determine potential resource bottleneck positions and obtain a storage space mapping; A data access channel determination module updates a cache access path based on the storage space mapping to determine a data access channel; A concurrent conflict coordination module detects and adjusts multi-thread conflicts in the data access channel to obtain a response time distribution; A process refining module tracks resource utilization based on the response time distribution, dynamically updates semantic parameters, and limits the adjustment amplitude to obtain a data processing instruction.

Images