Big data processing-oriented deep learning analysis method and system

Abstract

The invention relates to the technical field of deep learning, in particular to a deep learning analysis method and system for big data processing, and the method comprises the following steps: obtaining a node and port sequence to construct a path table, recognizing tensor merging nodes, dynamically extracting and updating nodes to analyze a delay sequence, and screening a main path mapping relation. And constructing a data flow diagram to generate an analysis result. According to the method, tensor combination nodes are extracted to construct a path fusion structure, structural state resetting is achieved by combining input data changes of training batches, path stability is analyzed and recognized through combination of a delay difference sequence and a peak sequence, main paths are screened with delay and resource parameters as standards, and a node chain relation is mapped; according to the method, a calculation graph sequence is subjected to visual archiving in a structured data flow form, so that comprehensive optimization of a structure evolution trend, path execution performance and a resource scheduling strategy in a large-scale heterogeneous training scene is realized, and the response adaptation capability of a training process in a dynamic data environment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a deep learning analysis method and system for big data processing. BACKGROUND

[0002] The technical field of deep learning belongs to the core branch of artificial intelligence, mainly studying how to simulate the human nervous system by constructing a multi-layer neural network model to realize automatic feature extraction and abstract expression of massive data, including convolutional neural network for image recognition, recurrent neural network for sequence modeling, generative adversarial network for sample generation, and variational autoencoder for probability modeling, covering model structure design, training strategy optimization, parameter adjustment mechanism and multi-task learning method, etc. Among them, the traditional deep learning analysis method for big data processing refers to the unified training of pre-set neural network structure to complete data feature extraction and pattern recognition when dealing with large volume, high dimension and complex heterogeneous data sets. Usually, multi-layer perceptron or deep convolutional neural network is used for modeling, and the back propagation algorithm is used to iteratively update the weight parameters based on large-scale sample set to obtain the model output result with strong feature expression ability.

[0003] The traditional deep learning analysis method uses pre-set neural network structure for unified training, relies on fixed model to complete feature extraction and pattern recognition in big data environment, lacks real-time perception ability of node combination relationship change, and is prone to path mismatch, structure instability or resource utilization imbalance when the difference of input data batches is high. For example, when part of the node combination mutates or the channel input characteristics change, the system cannot adjust the structure in time, resulting in training path blockage or performance degradation. At the same time, the dynamic correlation between path execution time and node behavior is not analyzed, it is difficult to effectively identify the key path of resource load concentration, and the stability and efficiency of the model training process are affected. SUMMARY

[0004] The purpose of the present application is to solve the problems existing in the prior art and to provide a deep learning analysis method for big data processing.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a deep learning analysis method for big data processing, comprising the following steps:

[0006] S1: Obtain the neural network graph structure node number and input port sequence, construct the path input source table, compare whether the source number and port combination are completely consistent, record the tensor merging node, and count the number of path fusion nodes;

[0007] S2: Based on the number of path fusion nodes, re-extract the current path input source table according to the new batch of training data, compare and analyze whether the combination of tensor merging nodes has changed, record the dynamically updated nodes, and generate a set of path update trigger nodes.

[0008] S3: Update the set of trigger nodes according to the path, count the fusion path structure number corresponding to the nodes that have not been refreshed, obtain the node timestamp to construct the delay time difference sequence and construct the peak sequence, calculate the difference and compare it with the judgment condition, record the low-latency stable path, and generate the stable path delay sequence.

[0009] S4: Based on the stable path delay sequence, collect the average path time parameters and resource allocation weights, extract the path number with the lowest time and resource weight coefficient above the median, perform node chain mapping within the training period, and generate the main path structure mapping result.

[0010] S5: Based on the main path structure mapping results, construct a structure data flow graph and perform layer-by-layer data flow link analysis, and archive the big data deep learning path analysis results.

[0011] As a further aspect of the present invention, the number of path fusion nodes includes a tensor fusion node set, node identification tags, and fusion path statistics; the path update trigger node set includes dynamically updated node identifiers, path number change records, and structural unit reset flags; the stable path delay sequence includes delay peak position numbers, delay time difference sequences, and stability judgment flags; the results of the main path structure mapping include a main path number set, resource utilization evaluation indicators, and node chain mapping relationships; and the results of the big data deep learning path analysis include a structural topology sequence, data flow graph node mapping results, and visualized archived data.

[0012] As a further aspect of the present invention, the step of obtaining the number of path fusion nodes specifically includes:

[0013] S111: Based on all node numbers and corresponding input port sequences in the neural network graph structure, extract the input port sequence of each node, and establish a path input source table composed of node numbers and input port sequences. Read the node information in the path input source table for identification and classification operations, perform unique mapping according to the combination of node numbers and port sequences, and construct a path mapping structure table accordingly.

[0014] S112: Based on each node record in the path mapping structure table, obtain the corresponding computation graph topology parameters, and read the number of input channels, the number of output channels, and the tensor shape in the topology parameters. Compare the consistency between the source number and the input port combination in the path input source table, filter the node combinations whose source numbers and port combinations are completely consistent, and obtain the path structure consistency matching result.

[0015] S113: Based on the node combination identified by the path structure consistency matching result, locate the corresponding node number, and count the number of nodes that meet the tensor merging conditions. By judging the consistency between its channel number and tensor shape, and combining the number of input port mapping combinations, the number of path fusion nodes is obtained.

[0016] As a further aspect of the present invention, the step of obtaining the path update trigger node set specifically includes:

[0017] S211: Based on the number of path fusion nodes, extract the current path input source table during the new batch training data loading stage, synchronize the structure with the node port sequence recorded in the previous batch path input source table, rearrange and index the path combination formed by the node number and its input port sequence, and obtain the current path input sequence matrix.

[0018] S212: Based on the current path input sequence matrix, compare the node numbers and input port sequence combinations in the previous batch of path input sequence matrices, perform an XOR operation on the hash value of the port sequence corresponding to any number, calculate the path combination change intensity value, and when it is greater than the preset structural difference threshold, record the node number and determine the dynamic structural change to obtain the dynamically updated node index set.

[0019] S213: Based on the dynamically updated node index set, read the computation status record structure of the structural unit where the node is located, identify the running identifier of the structural unit, obtain the corresponding container group number through the identifier matching operation, execute the structural unit status reset command, collect all node numbers and structural unit identifiers that trigger the reconstruction operation, and obtain the path update trigger node set.

[0020] As a further aspect of the present invention, the step of obtaining the stable path delay sequence specifically includes:

[0021] S311: Based on the path update trigger node set, count the node structure numbers that have not been reset in the current batch, obtain the corresponding fusion path structure index in the order of node numbers, associate it with the training timestamp records that appear in the current batch, record the mapping between structure number and timestamp value, and generate a node timestamp index mapping table.

[0022] S312: Read the node timestamp index mapping table, perform difference calculation on the timestamps between adjacent node numbers, construct a time difference sequence, set a sliding window to slide the entire sequence in sequence, compare the time difference of each node in each window, filter the node number with the largest difference, record the position information of the corresponding node number in the sequence, and combine the corresponding path number and timestamp position to form a set to obtain a continuous batch delay peak sequence.

[0023] S313: Based on the continuous batch delay peak sequence, using the batch index as a reference, calculate the position difference of the delay peak number in three consecutive batches. If the position difference is not greater than two, and at the same time, it is determined that the maximum time difference in the three batches does not exceed the threshold, filter the path number records that meet the conditions, and generate a stable path delay sequence.

[0024] As a further aspect of the present invention, the step of obtaining the main path structure mapping result specifically includes:

[0025] S411: Based on the stable path delay sequence, collect the time consumption record of each path in the training batch, count the difference between the path number and the training time of each batch, calculate the average time consumption parameter of each path, and at the same time collect the resource configuration record of each path in the corresponding batch, extract the resource allocation weight, and generate a path time consumption and resource index table.

[0026] S412: Based on the path time and resource index table, sort the average time values ​​of all paths in ascending order of numerical value, extract the path numbers whose resource allocation weight values ​​are greater than the median of resource weights, calculate the path optimization score, sort and filter the corresponding number set, and generate the main training path number list.

[0027] S413: Based on the main training path number list, according to the node index number recorded in the training batch for each number, establish a chain index relationship mapping between the node number and the path number within the path, mark the topological connection order between nodes and the corresponding training time sequence index field, and establish the main path structure mapping result.

[0028] As a further aspect of the present invention, the steps for obtaining the big data deep learning path analysis results are specifically as follows:

[0029] S511: Based on the main path structure mapping result, extract the node chain structure in each path, sort the node sequence according to the time index recorded by the node during training, determine whether the connection between each pair of adjacent nodes satisfies the computation logic of operation input dependency, mark compliant node segments, summarize the start and end node numbers, sequence order and path number of all compliant node segments under each path, and obtain a list of compliant computation graph node index segments.

[0030] S512: Based on the list of node index segments in the compliant computation graph, extract the training time series data and resource configuration data of each node structure in each path in sequence, calculate the training time difference between adjacent nodes, read the relative position sequence value between node structures and compare it with the resource configuration range of the connection segment to obtain the structure flow connection strength dataset.

[0031] S513: Based on the structured flow connection strength dataset, perform hierarchical analysis on the connection links between nodes in each path, aggregate analysis on the connection directionality, flow rate and resource usage change trend of the links, construct a structured data flow graph according to the arrangement order of nodes in the path structure graph, and archive the big data deep learning path analysis results.

[0032] A deep learning analytics system for big data processing includes:

[0033] The path node identification module obtains the node number and input port sequence, combines and compares the node topology parameters, determines the matching items and counts the number of matching nodes, and generates the number of path fusion nodes.

[0034] The state change detection module compares the number and port sequence in the path input source table based on the number of path fusion nodes, marks the difference nodes and resets the structure state, and generates a set of path update trigger nodes.

[0035] The delay stability identification module updates the set of trigger nodes according to the path, extracts the path number of the unrefreshed node and the timestamp of the continuous training data, constructs the delay difference and extracts the peak number sequence, filters low-latency stable paths, and generates a stable path delay sequence.

[0036] The main path construction and filtering module extracts path time and resource weights based on the stable path delay sequence, filters out paths that do not meet the median weight, establishes a numbered chain structure, and generates the main path structure mapping result.

[0037] The link structure archiving module extracts the node topology sequence and constructs a data flow graph based on the main path structure mapping result, analyzes the link structure layer by layer and archives it, and generates big data deep learning path analysis results.

[0038] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0039] In this invention, a path fusion structure is constructed by extracting tensor merging nodes, and the structure state is reset by combining the changes in the input data of the training batch. The stability of the path is identified by the combined analysis of the delay difference sequence and the peak sequence. The main path is selected and the node chain relationship is mapped by using delay and resource parameters as the standard. The computation graph sequence is visualized and archived in the form of a structured data stream. This enables comprehensive optimization of the structure evolution trend, path execution performance and resource scheduling strategy in large-scale heterogeneous training scenarios, and improves the response and adaptation capability of the training process in dynamic data environments. Attached Figure Description

[0040] Figure 1 This is a flowchart of the main steps of the present invention;

[0041] Figure 2This is a flowchart of the process for obtaining the number of path fusion nodes in this invention;

[0042] Figure 3 This is a flowchart illustrating the process of obtaining the set of trigger nodes for path update in this invention.

[0043] Figure 4 This is a flowchart of the stable path delay sequence acquisition process of the present invention;

[0044] Figure 5 This is a flowchart of the process for obtaining the main path structure mapping results of this invention;

[0045] Figure 6 This is a flowchart of the process for obtaining the results of big data deep learning path analysis in this invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0047] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0048] Please see Figure 1 A deep learning analysis method for big data processing includes the following steps:

[0049] S1: Obtain all node numbers and input port sequences of the neural network graph structure, construct a path input source table, read the computation graph topology parameters of all nodes in the path input source table (including the number of node input / output channels and tensor shape), compare whether the source number and port combination are completely consistent, if there is a consistent combination, record the corresponding node as a tensor merging node (conforming to the node fusion operation of the TensorFlow / PyTorch framework), and count the number of path fusion nodes;

[0050] S2: Based on the number of path fusion nodes, the current path input source table is re-extracted during the new batch training data loading stage. It is compared with the previous batch path input source table to analyze whether the combination of tensor merging nodes has changed. If there is a difference in any item in the path number or port sequence, it is recorded as a dynamic update node and the state of the structural unit is reset (the state is cleared by using Kubernetes pod reconstruction strategy) to generate a set of path update trigger nodes.

[0051] S3: Update the set of trigger nodes according to the path, count the fusion path structure number corresponding to the nodes that have not been refreshed, obtain the node timestamp record during training of each batch of data, construct the delay time difference sequence by node number order, extract the position number of the maximum delay value using a sliding window and form a peak sequence, perform difference calculation on the delay peak sequence in consecutive batches, when the difference of the delay peak position number in any three consecutive batches does not exceed 2 and the time difference is less than 20ms (20ms threshold: based on the SLA response time requirement of the real-time computing system), then record the path as a low-latency stable path (compliant with the time-sensitive network standard), and generate a stable path delay sequence;

[0052] S4: Based on the stable path delay sequence, collect the average time parameter and resource allocation weight of each path in each batch of training (using the vcore-memory weight allocation of the YARN resource scheduler), sort them in ascending order according to the average time parameter, extract the path number with the lowest time and the resource weight coefficient above the median, form the main training path set, and determine the node chain mapping relationship of the set in the training period to generate the main path structure mapping result;

[0053] S5: Based on the main path structure mapping results, determine the computation graph topology sequence of each main path within the training cycle (conforming to the ONNX computation graph representation standard), construct the structured data flow graph and perform layer-by-layer data flow link analysis (using the Netflix Vizceral visualization solution standard), perform archiving processing, and generate big data deep learning path analysis results.

[0054] The number of path fusion nodes includes tensor fusion node set, node identification labels, and fusion path statistics. The path update trigger node set includes dynamically updated node identifiers, path number change records, and structural unit reset flags. The stable path delay sequence includes delay peak position numbers, delay time difference sequences, and stability judgment flags. The results of the main path structure mapping include the main path number set, resource utilization evaluation indicators, and node chain mapping relationships. The results of the big data deep learning path analysis include structural topology sequences, data flow graph node mapping results, and visualized archived data.

[0055] Please see Figure 2 Step S1 is as follows:

[0056] S111: Based on all node numbers and corresponding input port sequences in the neural network graph structure, extract the input port sequence of each node, and establish a path input source table composed of node numbers and input port sequences. Read the node information in the path input source table for identification and classification operations, perform unique mapping according to the combination of node numbers and port sequences, and construct a path mapping structure table accordingly.

[0057] Based on the node numbers and input port sequences in the neural network graph structure, it is necessary to extract the basic information of each node in the neural network graph structure item by item, and construct a path input source table in combination with the actual computation process of the neural network. In practice, neural networks are often represented by graph structures. For example, in a convolutional neural network, each layer of operation node has a unique number. For example, nodes N1, N2, N3, and N4 represent convolution operations or activation operations at different levels, respectively. The input port sequence can be obtained by calling the computation graph connection structure. For example, if node N3 receives input from both N1 and N2, its input port sequence is [1, 2]. In the data organization process, the node number and its input port are mapped one-to-one to form a key-value mapping relationship, thereby generating a path input source table. By uniformly traversing and mapping all nodes, the following path mapping structure table is constructed:

[0058] Table 1 Path Mapping Structure

[0059]

[0060] As shown in Table 1, nodes N3 and N4 have multiple input ports. Their structure is suitable for subsequent path matching analysis operations. The path input source table is standardized by port sequence (ascending sort, sequence uniqueness, etc.), and then grouped and mapped by combining node numbers and port sequences to complete the construction of the path mapping structure table. This facilitates subsequent comparison and filtering of information such as node numbers and connection topology, and finally generates the path mapping structure table.

[0061] S112: Based on each node record in the path mapping structure table, obtain the corresponding computation graph topology parameters, and read the number of input channels, number of output channels, and tensor shape in the topology parameters. Compare the consistency between the source number and the input port combination in the path input source table, filter the node combinations whose source numbers and port combinations are completely consistent, and obtain the path structure consistency matching result.

[0062] After obtaining the path mapping structure table, topology parameter parsing is required for each node. The system reads the corresponding number of input channels, number of output channels, and tensor dimensions for each node. The shape of the tensor dimension is obtained by multiplying each dimension. For example, if a node's output tensor is [32, 16, 16], its shape product is 8192, which is used as a quantification index of the tensor shape. Further comparison is made between the source number and input port combination in the path input source table to see if there is a consistent combination. For example, if both node N2 and node N3 have input ports [1, 2], then it is considered a structural match. The matching criterion is that the port combination is completely identical, regardless of the port order. A set comparison method is used for judgment. In actual analysis, the system uses this matching mechanism to filter and obtain the following topology parameter illustration data:

[0063] Table 2. Comparison of Topology Parameters

[0064]

[0065] As shown in Table 2, nodes N2 and N3 have the same input port combination. Their source number is identified as consistent with the input port combination, and they are determined to be path structure consistency matching items. Through this item-by-item comparison method, a total of 2 structurally consistent combinations were identified, namely N2-N3 and N1-N2, and the path structure consistency matching result was finally obtained.

[0066] S113: Based on the node combination identified by the path structure consistency matching result, locate the corresponding node number and count the number of nodes that meet the tensor merging conditions. By judging the consistency between its channel number and tensor shape, and combining the number of input port mapping combinations, the number of path fusion nodes is obtained.

[0067] Based on the path structure consistency matching results, a fusion condition judgment operation is performed on each node. Specifically, this requires calling the number of input channels, the number of output channels, and the tensor dimension product of the aforementioned nodes, and combining this with the number of repeated input port mappings for condition filtering. The fusion threshold range is set as follows: the difference between input and output channels does not exceed 128, the tensor dimension product does not exceed 65536, and the number of repeated input port mappings does not exceed 3 times. During the judgment process, the system performs the following operations: Node N1 has 32 input channels, 64 output channels, a difference of 32, and a tensor product of 819. 2. The repetition count is 1, which satisfies all conditions. Node N2 has 64 channels and a tensor product of 16384, with a repetition count of 1, which also satisfies the conditions. Node N3 has 64 inputs and 128 outputs, with a difference of 64 and a tensor product of 32768, with a repetition count of 2, which meets the set restrictions. Although node N4 satisfies the first two conditions, its repetition count reaches 3, which is a critical boundary. It can be excluded based on the actual network situation. During this process, the system counts that there are 3 nodes that meet all fusion conditions, namely nodes N1, N2, and N3, and finally obtains the number of path fusion nodes.

[0068] Please see Figure 3 Step S2 is as follows:

[0069] S211: Based on the number of path fusion nodes, extract the current path input source table during the new batch training data loading stage, synchronize the structure with the node port sequence recorded in the previous batch path input source table, rearrange and index the path combination formed by the node number and its input port sequence, and obtain the current path input sequence matrix.

[0070] Based on the number of path fusion nodes, the node numbers and their corresponding input port structures in the new batch of training data loading stage are first obtained. By performing topological traversal on the input edges of each node, its input source number is identified and the ports are arranged according to the input index. For example, the input port of node N1 is [0], indicating that it only accepts the output of the previous layer; the input port of node N2 is [1], node N3 is [1, 2], and node N4 is [2, 3]. This structure is obtained based on the node connection table in the graph structure record. After collection, the nodes are arranged in ascending order. The path number and the input port sequence are mapped by a double index and recorded as the current path input source structure matrix. This matrix serves as the basis for structure comparison. After completion, the path input structure stored in the previous batch is called, and the node number alignment and input port sequence standardization operations are performed. In the case of disordered port combination (such as [2, 1]), ascending order processing is required to generate the standard combination [1, 2]. After standardization, the path structure of all nodes in the path combination is relabeled to form a unified current path input sequence matrix. This structure is used for the next step of structure change analysis. Table 2 shows the collection results of the input structure of each node in this stage:

[0071] Table 3 Current Batch Path Input Structure Table

[0072]

[0073] As shown in Table 3, the multi-input combination structure of nodes N3 and N4 has a key impact on the identification of subsequent combination changes. Finally, the current path input sequence matrix is ​​generated through input port structure reconstruction and standard index establishment.

[0074] S212: Based on the current path input sequence matrix, compare the node numbers and input port sequence combinations in the previous batch of path input sequence matrices, and perform an XOR operation on the hash values ​​of the port sequences corresponding to any number, using the formula:

[0075]

[0076] Calculate the intensity value of path combination change. When it exceeds a preset structural difference threshold, record the node number and determine the dynamic structural change, thus obtaining a dynamically updated node index set; where S′ diffThis represents the structural change trigger strength value, expressed in hash units, where m represents the number of path fusion nodes. and L represents the hash value of the combination of the input ports of the i-th node in the current batch and the previous batch, respectively, in units of hash integers. i The input link hop count for the i-th node is represented by ΔT. i R represents the time difference between the current and previous batches when the i-th node is extracted from the path input table, in milliseconds. i This represents the count value of the historical repeated input port structure of this node, log2(1+R). i ) represents the repetitive adjustment function term;

[0077] After calling the current path input sequence matrix, the input port combination of each node in the current batch is compared with the data of the previous batch. The combination feature is represented by structural hashing. For example, the combination [1, 2] is generated into a hash value through weighted mapping. The current hash value of node N1 is 214, and the previous one was 212. The difference is 2, indicating that its structure has changed slightly. The hash value difference of node N3 is 8, which is a strong structural change. In addition, the path hop count (number of layers across the link), path extraction time interval (in milliseconds), and historical repetition count need to be collected to construct the change intensity value. This process requires analysis of the path topology of the nodes and collection of hop count information for each path. For example, N3 has 4 layers of deep input hop count. The extraction time difference between nodes is the difference between the timestamps of the two path data export records recorded by the system. The repetition count is the frequency of the current node structure in the previous batch. All parameter sampling is summarized in Table 4.

[0078] Table 4 Path Dynamic Update Parameter Table

[0079]

[0080] Based on the node parameters collected in Table 4, multi-level calculations are performed by substituting them into the formula:

[0081] N1 item:

[0082] N2 item:

[0083] N3 item:

[0084] N4 item:

[0085] S′ diff =7 + 9.36 + 17.79 + 15.86 = 49.99;

[0086] The calculation result will be compared with the threshold δ = 32. This threshold is set based on the number of path fusion nodes m = 4 in the current neural network structure, the maximum possible value of the path hop count being 5 layers, the maximum input port acquisition period being set to 40ms, and the theoretical upper limit of the structural variation intensity of the parameter combination being 64. Half of this value is taken as the structural variation threshold. Under the above data conditions, the calculated result is approximately 32. Therefore, a threshold δ = 32 is set. This threshold increases with the increase of the number of nodes m, the increase of the number of path hops L, or the increase of the input frequency (ΔT decreases). This reflects the correlation between the intensity of structural changes and the system scale and input density, and has a complete logical derivation basis and data attribution conditions. Because S′ diff >δ, the system determines that the current batch path structure combination has undergone structural changes, and the structure update logic needs to be triggered to obtain a dynamically updated node index set.

[0087] The path combination change intensity value represents the overall change in the input path structure combination of each node in the neural network between two consecutive training batches. This value is calculated by measuring the difference in input port hash encoding, path hop count, and input sampling time interval of each node, combined with the historical repetition of the node structure. The larger the value, the more frequent or greater the dynamic changes in the node's structural topology. It is an important indicator reflecting the stability of the neural network structure and the trend of path dynamic reconstruction during training. This intensity value not only reflects the increase in the complexity of the input path at the topological level, but also reveals the concentration and activity of structural adjustments between the current batch and the previous batch. It can be used to determine whether a container-level structural reset operation needs to be performed.

[0088] The formula is based on the idea of ​​comprehensively measuring the multi-dimensional characteristics of path structure changes, where the first part of the term... The hash difference representing the current node's input port structure between two batches serves as the most direct encoding feature of path changes. Using absolute value operations avoids directional interference and retains only the intensity of the change; the second part... It is a joint representation of path hop count, input extraction time interval, and historical repetition, where L i With ΔT i These respectively reflect the path topology complexity and update cycle density; their product is used to quantify the cumulative effects that structural changes may cause. Introducing nonlinear scaling in the form of square roots reduces the dominance of extreme values, while the log2(1+R) in the denominator... iThe weighting of repeated structures is used to suppress the influence of repeated path combinations in change evaluation, thereby reducing the impact of repeated path combinations in change evaluation and enhancing the stability of the formula in scenarios with frequent structural reconstruction. The overall structure adopts additive combination to ensure that hash difference and path dynamic features participate in the cumulative evaluation of change intensity, so that the formula achieves a dynamic balance between the frequency of structural changes and topological complexity.

[0089] S213: Based on the dynamically updated node index set, read the calculation status record structure of the structural unit where the node is located, identify the running identifier of the structural unit, obtain the corresponding container group number through the identifier matching operation, execute the structural unit status reset command, collect all node numbers and structural unit identifiers that trigger the reconstruction operation, and obtain the path update trigger node set.

[0090] Based on the node numbers recorded in the dynamically updated node index set, the current structural unit instance of each node is queried sequentially. The unique identifier of the corresponding container group Pod is obtained through the container orchestration system. For example, the Pod identifier of node N2 is pod-n2-t3. After identification, the state of these structural units is reset, Pod-level cleanup instructions are executed and their state is changed to Pending, waiting for the container to be rescheduled to generate a new state. During this process, the system automatically clears the running configuration, memory cache and intermediate state data. After reconstruction, the Pod container corresponding to each node will generate a new ID identifier for subsequent scheduling records. Finally, all reconstructed node numbers and their structural unit identifiers are summarized to generate a structural update event index and obtain the set of path update trigger nodes.

[0091] Please see Figure 4 Step S3 is as follows:

[0092] S311: Based on the set of path update trigger nodes, count the node structure numbers that have not been reset in the current batch, obtain the corresponding fusion path structure index in the order of node numbers, associate it with the training timestamp records that appear in the current batch, record the mapping between structure number and timestamp value, and generate a node timestamp index mapping table.

[0093] Based on the set of path update trigger nodes, the structure numbers of nodes whose states have not been reset in the current batch are obtained sequentially. All active nodes are filtered out through the node running status record structure. For each node, the structure number index is called to obtain its corresponding fusion path structure number. The timestamp records of the nodes participating in the training process in the current batch are matched using an index mapping mechanism. For example, nodes N1 and N2 start training at 1020ms and 1032ms of the batch, respectively; nodes N3 and N4 appear at 1048ms and 1060ms, respectively; and node N5 appears at 1078ms. The node number, corresponding path structure number, and timestamp value are combined to construct a two-dimensional structure table mapping, thus obtaining the node number to timestamp mapping table structure. In the mapping structure, the number and timestamp information of multiple nodes corresponding to all path structure numbers are independent of each other, and are arranged in ascending order of structure number and ascending order of node number using a dual index order. This index mechanism can be used for subsequent delayed sequence extraction operations to generate a node timestamp index mapping table, as shown in Table 5.

[0094] Table 5 Node Timestamp Index Mapping Table

[0095]

[0096]

[0097] As shown in Table 5, nodes N1 and N2 belong to path P1, N3 and N4 belong to path P2, and N5 belongs to path P3. The distribution of node timestamps reflects the order of the batches when the nodes start executing.

[0098] S312: Read the node timestamp index mapping table, perform difference calculation on the timestamps between adjacent node numbers, construct a time difference sequence, set a sliding window to slide the entire sequence in turn, compare the time difference of each node in each window, filter the node number with the largest difference, record the position information of the corresponding node number in the sequence, and combine the corresponding path number and timestamp position to form a set to obtain a continuous batch delay peak sequence.

[0099] Read the node timestamp index mapping table, traverse the node number sequence sequentially, extract the timestamp values ​​of adjacent nodes, and construct a difference array according to the number order. After construction, set the window length to 5. Within this window, slide 1 position to the right each time to extract window segment data. Extract the 5 timestamp subsequences contained in each window segment, and compare the 4 time differences in each subsequence. For example, in the window [N1, N2, N3, N4, N5], the timestamps are 1020ms, 1032ms, 1048ms, 1060 ... 78ms, the adjacent differences are calculated to be 12ms, 16ms, 12ms, and 18ms. The maximum time difference in this subsequence is 18ms, corresponding to position N5. This node number and the structure path number P3 are recorded together as the delay peak point of this window segment. Then the window is moved to the right. If there are more nodes, the operation is repeated. In this batch example, only one set of peak records can be obtained. After marking the peak node, the node number, delay value and corresponding path number are extracted as peak items and added to the sequence record to obtain a continuous batch delay peak sequence.

[0100] S313: Based on the continuous batch delay peak sequence, using the batch index as a reference, calculate the position difference of the delay peak number in three consecutive batches. If the position difference is not greater than two, and at the same time, it is determined that the maximum time difference in the three batches does not exceed the threshold, filter the path number records that meet the conditions, and generate a stable path delay sequence.

[0101] Based on the consecutive batch delay peak sequence, the current batch is compared with the delay peak records of the previous two batches. The peak node number of each record in each batch is called one by one, and the absolute difference of the three numbers is calculated. If the difference between any two of them does not exceed 2, and the maximum value of the delay value field in the three records is judged, and the maximum delay difference does not exceed 20ms, then the batch delay peak consistency requirement is met. This judgment standard is the stability judgment threshold. The reference value of 20ms is derived from the Service Level Agreement (SLA) limit standard for path response delay of the real-time computing system. This standard is derived from the mean and variance calculation results of the path response time distribution collected by the system operation monitoring module. According to historical operation monitoring statistics, in response events within 50ms, the effective acceptable jitter does not exceed 20ms, which is used as the stable path identification benchmark. In the above judgment, if the delay peak position change of a certain path number in the three batches is less than or equal to 2, and the maximum delay value is less than or equal to 20ms, then the path number is recorded as a stable path. Its fused path structure number and delay value index are extracted and recorded in the stable path record structure, and finally a stable path delay sequence is generated.

[0102] Please see Figure 5 Step S4 is as follows:

[0103] S411: Based on the stable path delay sequence, collect the time consumption records of each path in the training batch, count the difference between the path number and the training time of each batch, calculate the average time consumption parameter of each path, and at the same time collect the resource configuration records of each path in the corresponding batch, extract the resource allocation weight, and generate a path time consumption and resource index table.

[0104] Based on the stable path delay sequence, average latency and resource allocation data are collected sequentially for each path. First, the start and end timestamps corresponding to each path across all training batches are extracted. The latency value for each batch is calculated. Then, these values ​​are summed and divided by the batch number to obtain the average latency value corresponding to the path number. For example, if the training latency of path P1 in the five batches is 1030ms, 1060ms, 1045ms, 1065ms, and 1050ms respectively, then its average latency value is (1030+1060+1045+1065+1050) / 5 = 1050ms. The resource allocation records for each path in each batch are parsed. The number of vcores and the corresponding memory capacity allocated by the YARN scheduling module are converted into a unified dimension, and their product is taken as the base value of the resource weight. The average value is calculated for the same batch to obtain the resource allocation weight value corresponding to the path number. Then, the resource cost value of each path is calculated. The calculation method is the product of the average time and the resource weight value. For example, if the average time of path P1 is 1050ms and the resource allocation is 2.0 vcores, then the resource cost is 1050 × 2.0 = 2100ms·vcore. The results are shown in Table 6.

[0105] Table 6. Path Resource Scheduling and Time Consumption Indicators

[0106]

[0107] As shown in Table 6, path P3 has the lowest resource cost, while path P5 has the highest resource cost. These data will serve as important inputs for path optimization, generating a table of path time and resource metrics.

[0108] S412: Based on the path time and resource index table, sort all paths by average time value in ascending order, extract the path numbers whose resource allocation weight value is greater than the median resource weight, using the formula:

[0109]

[0110] Calculate the path optimization score, sort and filter the corresponding number set, and generate a list of main training path numbers; where R′ gj This represents the path g relative to the median resource cost O. m The preferred score, T g W represents the average time taken for path g, in milliseconds (ms).g This represents the resource allocation weight of path g, expressed in virtual cores (vcores), O g This represents the resource sales volume for path g, calculated as O. g =T g ·W g The unit is milliseconds multiplied by the number of cores (ms·vcore), O m This represents the median resource cost across all paths, expressed in milliseconds (ms·vcore). T represents the summation operation for each path k in the path set from 1 to n. k W k O k These represent the average time, resource weight, and resource cost of path k, respectively, with units consistent with the above.

[0111] Based on the path time and resource index table, all path numbers are evaluated for optimal performance. First, the "Resource Cost" column in the table is sorted in ascending order to obtain the median value of O. m =2712.5ms·vcore. Paths with resource overhead less than this value, namely P1 and P3, are selected for optimization scoring. Taking path P1 as an example, its average time T... g =1050ms, resource weight W g =2.0vcore, resource overhead O g =2100ms·vcore, substituting into the formula:

[0112] The molecule is:

[0113]

[0114] The denominator is:

[0115]

[0116] Right now:

[0117] 1 + 525.0 + 746.7 + 326.7 + 650.0 + 770.0 = 3019.4;

[0118] The final score is:

[0119]

[0120] Performing the same calculation on path P3 yields:

[0121]

[0122] molecular:

[0123]

[0124] The denominator is the same as above, which is 3019.4. The score is:

[0125]

[0126] Based on the scores, path P3 is superior to P1 and has a higher score. Path P3 is ultimately selected as the best choice, resulting in a list of main training path numbers.

[0127] The path optimization score is a numerical indicator used to measure the relative performance of each training path under multiple dimensions such as average time consumption, resource allocation intensity, and system resource scheduling consistency. This score comprehensively reflects the combined effect of the deviation in resource utilization efficiency and time consumption performance of the path. The smaller the score, the more the path has a resource consumption state that is closer to the median level of the overall system scheduling, while having a more reasonable time consumption performance, and is therefore considered better in path selection. In other words, the smaller the path optimization score, the more the path has both high resource coordination and high time efficiency in the training task, and is more suitable to be included in the main training path set and participate in the subsequent chain structure mapping as a scheduling priority path.

[0128] The formula is based on a comprehensive evaluation principle of resource efficiency and relative deviation. First, in the numerator, the average path time T is calculated. g Its allocated resource weight W g The square roots of the values ​​are added together to represent the primary performance indicator for evaluating path scheduling efficiency. The resource weights, after being square-rooted, are added to reflect the impact of resource allocation but avoid exponentially dominating the overall score, thus reflecting the secondary impact of resource scheduling on path efficiency. This sum is then divided by the ratio of the resource cost deviation. Multiplication is used to amplify the score differences of paths that, while having reasonable time consumption, deviate significantly from the median of the overall system resource allocation, thereby eliminating paths with abnormal resource utilization during the optimization process; in the denominator, a weighted normalized denominator is constructed. It reflects the coordination between resource consumption and time consumption of all paths, with the unit structure being ms / vcore. It can measure whether each path has a time efficiency imbalance caused by resource redundancy or insufficient resources. The constant term 1 is used to avoid the denominator being 0 and to maintain the stability of the scoring mechanism. Finally, the entire formula forms a four-dimensional coupled structure of "time consumption - resources - deviation - normalization", so that the scoring not only considers the characteristics of a single path, but also combines the group difference benchmark to quantitatively screen the path performance.

[0129] S413: Based on the main training path number list, according to the node index number recorded in the training batch for each number, establish a chain index relationship mapping between the node number and the path number within the path, mark the topological connection order between nodes and the corresponding training time sequence index field, and establish the main path structure mapping result.

[0130] Based on the main training path number list, taking P3 as an example, extract the sequence of node numbers associated with it. Read the node numbers from the training scheduling record and arrange them in chronological order as N2→N4→N7→N9. Set the chain connection order as N2→N4, N4→N7, N7→N9. Record the connection relationship between each pair of nodes, the relative time difference in the training cycle, and the path number P3 to which they belong, forming a node chain structure. Then, merge and collect the chain structures in all main training paths. Construct a mapping table structure with the path number as the key and the chain node sequence as the value. Record the node chain structure of path P3 as {P3: [(N2, N4), (N4, N7), (N7, N9)]}. Finally, output and archive this mapping structure for use in the subsequent generation of scheduling instructions for the training system, and establish the main path structure mapping result.

[0131] Please see Figure 6 The S5 steps are as follows:

[0132] S511: Based on the main path structure mapping results, extract the node chain structure in each path, sort the node sequence according to the time index recorded by the node during training, determine whether the connection between each pair of adjacent nodes satisfies the computation logic of operation input dependency, mark compliant node segments, summarize the start and end node numbers, sequence order and path number of all compliant node segments under each path, and obtain the list of compliant computation graph node index segments.

[0133] Based on the main path structure mapping result, the node chain structure contained in each path is obtained sequentially. The number information, connection order index, and path number of each pair of connected nodes in the structure are extracted as basic structure fields. By reading the temporal index of each node connection segment, the node pairs are arranged in ascending order according to the index value to achieve structure reconstruction in the training time dimension. At the same time, the upstream and downstream relationship of each pair of node connections is judged, that is, whether the starting node is the input source node of the downstream node is detected, and further, whether the connection segment has non-linear loop connection or skip connection behavior is read. For example, in path P1, node N1 connects to N2, and N2 connects to N3. If N3 is found to be directly connected to N1, it forms a loop chain connection, which does not meet the linear topology standard. Then, the connection structure that meets the continuous input logic is marked, and the starting node and ending node of the node chain structure that continuously satisfies the linear dependency condition in each path are paired and recorded. Finally, a path number, structure segment index range and topology order list structure is established, as shown in Table 7. The table uses path numbers P1, P2 and P3 as examples to list the connection order, training time and resource configuration in each path, which are used to judge the structural coherence and temporal consistency.

[0134] Table 7 Node Connection Structure and Performance Parameters

[0135]

[0136] As shown in Table 7, the node connection structure of path P1 satisfies the linear topological relationship, the training time shows an increasing trend, the resource allocation has no obvious jump, which meets the requirements of the ONNX computation graph format, and finally generates a list of compliant computation graph node index segments.

[0137] S512: Based on the list of node index segments in the compliant computation graph, extract the training time series data and resource configuration data of each node structure in each path in sequence, calculate the training time difference between adjacent nodes, read the relative position sequence value between node structures and compare it with the resource configuration range of the connection segment to obtain the structure flow connection strength dataset.

[0138] Based on the list of node index segments in the compliant computing graph, the training time and resource allocation data of the connections between nodes in each segment are extracted one by one. The time difference and resource change magnitude between each pair of connected nodes are calculated sequentially. The consistency of the connection direction is determined by calling the node connection order. Then, the fluctuation range of resource configuration changes in the node connection sequence is extracted, and the average value of the fluctuation range is calculated using the absolute value of the difference. For example, in path P1, the training time for N1→N2 is 60ms, and the resource is 2vcore; the training time for N2→N3 is 80ms, and the resource is 3vcore. Therefore, the time difference is 20ms, and the resource difference is 1vcore. Subsequently, the training time for N3→N4 is 90ms, and the resource is 2vcore. Therefore, the difference between the previous... The resource difference is -1 vcore, and the total resource fluctuation range of the connected segments in path P1 is 1+1=2 vcore. The time fluctuation values ​​are 20ms and 10ms. The same calculation is performed on all connected segments in each path, and segments with a resource fluctuation range exceeding 2.5 vcore or a time increase exceeding 35ms are defined as abnormal segments of structural fluctuation. For example, in path P3, the resource difference of the N5→N6 connected segment is 0 vcore, but the time difference is 10ms, which is not exceeded. The N7→N8 segment is an independent link and lacks a complete upstream and downstream structure, so it is not included. Finally, the path number, connection order, resource fluctuation and time difference records are called to construct a set of structural connectivity and performance fluctuation evaluation values ​​and generate a structural flow connection strength dataset.

[0139] S513: Based on the structured flow connection strength dataset, perform hierarchical analysis on the connection links between nodes in each path, aggregate analysis on the connection directionality, flow rate and resource usage change trend of the links, construct a structured data flow graph according to the arrangement order of nodes in the path structure graph, and archive the big data deep learning path analysis results.

[0140] Based on the structured flow connection strength dataset, the node structure and corresponding training performance indicators of each connection segment are loaded for each path. Node number pairs, connection direction identifiers, time consumption information, and resource usage information are extracted from each segment. All node connection pairs are hierarchically aggregated according to path number, and a link structure graph is generated based on connection direction as the grouping criterion. Then, the number of node connections in each link segment, the difference in training time consumption between adjacent nodes, and the average utilization rate per unit resource are calculated. All link structures are aggregated according to the training time consumption distribution density. Link structures with a time consumption density in the range [60ms, 90ms] and a resource fluctuation rate between 0.5 and 1.5 vcores are identified as visual marker units. The node number sequence and direction relationship of this type of link are output. Multiple links are constructed and incorporated into the overall data flow structure graph. This structure graph and node performance annotations are written into an archived data table to obtain the results of big data deep learning path analysis.

[0141] A deep learning analytics system for big data processing includes:

[0142] The path node identification module obtains the node number and input port sequence, combines and compares the node topology parameters, determines the matching items and counts the number of matching nodes, and generates the number of path fusion nodes.

[0143] The state change detection module compares the number and port sequence in the path input source table with the number of path fusion nodes, marks the difference nodes and resets the structure state, and generates a set of path update trigger nodes based on the number of path fusion nodes.

[0144] The delay stability identification module updates the set of trigger nodes based on the path, extracts the path number of the unrefreshed node and the timestamp of the continuous training data, constructs the delay difference and extracts the peak number sequence, filters low-latency stable paths, and generates a stable path delay sequence.

[0145] The main path construction and filtering module extracts path time and resource weights based on stable path delay sequences, filters out paths that do not meet the median weight, establishes a numbered chain structure, and generates the main path structure mapping result.

[0146] The link structure archiving module extracts the node topology sequence and constructs a data flow graph based on the main path structure mapping results. It analyzes the link structure layer by layer and archives the data to generate big data deep learning path analysis results.

[0147] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A deep learning analysis method for big data processing, characterized in that, Includes the following steps: S1: Obtain the node numbers and input port sequences of the neural network graph structure, construct the path input source table, compare whether the source numbers and port combinations are completely consistent, record the tensor merging nodes, and count the number of path fusion nodes. S2: Based on the number of path fusion nodes, re-extract the current path input source table according to the new batch of training data, compare and analyze whether the combination of tensor merging nodes has changed, record the dynamically updated nodes, and generate a set of path update trigger nodes. S3: Update the set of trigger nodes according to the path, count the fusion path structure number corresponding to the nodes that have not been refreshed, obtain the node timestamp to construct the delay time difference sequence and construct the peak sequence, calculate the difference and compare it with the judgment condition, record the low-latency stable path, and generate the stable path delay sequence. S4: Based on the stable path delay sequence, collect the average path time parameters and resource allocation weights, extract the path number with the lowest time and resource weight coefficient above the median, perform node chain mapping within the training period, and generate the main path structure mapping result. S5: Based on the main path structure mapping results, construct a structure data flow graph and perform layer-by-layer data flow link analysis, and archive the big data deep learning path analysis results.

2. The deep learning analysis method for big data processing according to claim 1, characterized in that, The number of path fusion nodes includes tensor fusion node set, node identification tags, and fusion path statistics; the set of path update triggering nodes includes dynamically updated node identifiers, path number change records, and structural unit reset flags; the stable path delay sequence includes delay peak position numbers, delay time difference sequences, and stability judgment flags; the results of the main path structure mapping result include main path number set, resource utilization evaluation index, and node chain mapping relationship; and the results of the big data deep learning path analysis result include structural topology sequence, data flow graph node mapping results, and visualized archived data. The determination condition is: when the difference between the position numbers of the delay peak in any three consecutive batches does not exceed 2 and the time difference is less than 20ms.

3. The deep learning analysis method for big data processing according to claim 1, characterized in that, The specific steps for obtaining the number of path fusion nodes are as follows: S111: Based on all node numbers and corresponding input port sequences in the neural network graph structure, extract the input port sequence of each node, and establish a path input source table composed of node numbers and input port sequences. Read the node information in the path input source table for identification and classification operations, perform unique mapping according to the combination of node numbers and port sequences, and construct a path mapping structure table accordingly. S112: Based on each node record in the path mapping structure table, obtain the corresponding computation graph topology parameters, and read the number of input channels, the number of output channels, and the tensor shape in the topology parameters. Compare the consistency between the source number and the input port combination in the path input source table, filter the node combinations whose source numbers and port combinations are completely consistent, and obtain the path structure consistency matching result. S113: Based on the node combination identified by the path structure consistency matching result, locate the corresponding node number, and count the number of nodes that meet the tensor merging conditions. By judging the consistency between its channel number and tensor shape, and combining the number of input port mapping combinations, the number of path fusion nodes is obtained.

4. The deep learning analysis method for big data processing according to claim 1, characterized in that, The specific steps for obtaining the set of path update trigger nodes are as follows: S211: Based on the number of path fusion nodes, extract the current path input source table during the new batch training data loading stage, synchronize the structure with the node port sequence recorded in the previous batch path input source table, rearrange and index the path combination formed by the node number and its input port sequence, and obtain the current path input sequence matrix. S212: Based on the current path input sequence matrix, compare the node numbers and input port sequence combinations in the previous batch of path input sequence matrices, perform an XOR operation on the hash value of the port sequence corresponding to any number, calculate the path combination change intensity value, and when it is greater than the preset structural difference threshold, record the node number and determine the dynamic structural change to obtain the dynamically updated node index set. S213: Based on the dynamically updated node index set, read the computation status record structure of the structural unit where the node is located, identify the running identifier of the structural unit, obtain the corresponding container group number through the identifier matching operation, execute the structural unit status reset command, collect all node numbers and structural unit identifiers that trigger the reconstruction operation, and obtain the path update trigger node set.

5. The deep learning analysis method for big data processing according to claim 1, characterized in that, The specific steps for obtaining the stable path delay sequence are as follows: S311: Based on the path update trigger node set, count the node structure numbers that have not been reset in the current batch, obtain the corresponding fusion path structure index in the order of node numbers, associate it with the training timestamp records that appear in the current batch, record the mapping between structure number and timestamp value, and generate a node timestamp index mapping table. S312: Read the node timestamp index mapping table, perform difference calculation on the timestamps between adjacent node numbers, construct a time difference sequence, set a sliding window to slide the entire sequence in sequence, compare the time difference of each node in each window, filter the node number with the largest difference, record the position information of the corresponding node number in the sequence, and combine the corresponding path number and timestamp position to form a set to obtain a continuous batch delay peak sequence. S313: Based on the continuous batch delay peak sequence, using the batch index as a reference, calculate the position difference of the delay peak number in three consecutive batches. If the position difference is not greater than two, and at the same time, it is determined that the maximum time difference in the three batches does not exceed the threshold, filter the path number records that meet the conditions, and generate a stable path delay sequence.

6. The deep learning analysis method for big data processing according to claim 1, characterized in that, The specific steps for obtaining the main path structure mapping result are as follows: S411: Based on the stable path delay sequence, collect the time consumption record of each path in the training batch, count the difference between the path number and the training time of each batch, calculate the average time consumption parameter of each path, and at the same time collect the resource configuration record of each path in the corresponding batch, extract the resource allocation weight, and generate a path time consumption and resource index table. S412: Based on the path time and resource index table, sort the average time values ​​of all paths in ascending order of numerical value, extract the path numbers whose resource allocation weight values ​​are greater than the median of resource weights, calculate the path optimization score, sort and filter the corresponding number set, and generate the main training path number list. S413: Based on the main training path number list, according to the node index number recorded in the training batch for each number, establish a chain index relationship mapping between the node number and the path number within the path, mark the topological connection order between nodes and the corresponding training time sequence index field, and establish the main path structure mapping result.

7. The deep learning analysis method for big data processing according to claim 1, characterized in that, The specific steps for obtaining the big data deep learning path analysis results are as follows: S511: Based on the main path structure mapping result, extract the node chain structure in each path, sort the node sequence according to the time index recorded by the node during training, determine whether the connection between each pair of adjacent nodes satisfies the computation logic of operation input dependency, mark compliant node segments, summarize the start and end node numbers, sequence order and path number of all compliant node segments under each path, and obtain a list of compliant computation graph node index segments. S512: Based on the list of node index segments in the compliant computation graph, extract the training time series data and resource configuration data of each node structure in each path in sequence, calculate the training time difference between adjacent nodes, read the relative position sequence value between node structures and compare it with the resource configuration range of the connection segment to obtain the structure flow connection strength dataset. S513: Based on the structured flow connection strength dataset, perform hierarchical analysis on the connection links between nodes in each path, aggregate analysis on the connection directionality, flow rate and resource usage change trend of the links, construct a structured data flow graph according to the arrangement order of nodes in the path structure graph, and archive the big data deep learning path analysis results.

8. A deep learning analysis system for big data processing, characterized in that, The system is used to implement the deep learning analysis method for big data processing as described in any one of claims 1-7, comprising: The path node identification module obtains the node number and input port sequence, combines and compares the node topology parameters, determines the matching items and counts the number of matching nodes, and generates the number of path fusion nodes. The state change detection module compares the number and port sequence in the path input source table based on the number of path fusion nodes, marks the difference nodes and resets the structure state, and generates a set of path update trigger nodes. The delay stability identification module updates the set of trigger nodes according to the path, extracts the path number of the unrefreshed node and the timestamp of the continuous training data, constructs the delay difference and extracts the peak number sequence, filters low-latency stable paths, and generates a stable path delay sequence. The main path construction and filtering module extracts path time and resource weights based on the stable path delay sequence, filters out paths that do not meet the median weight, establishes a numbered chain structure, and generates the main path structure mapping result. The link structure archiving module extracts the node topology sequence and constructs a data flow graph based on the main path structure mapping result, analyzes the link structure layer by layer and archives it, and generates big data deep learning path analysis results.

Images