System and method for extracting multi-sentence features

Abstract

A system for analyzing a technical document of a storage device and extracting a multi-sentence feature is disclosed. The system includes: a plurality of classifiers, each classifier configured to receive a plurality of sentences from a technical document and generate tags for the plurality of sentences, each of the plurality of tags indicating whether each sentence has a target feature described in the technical document; and a combined neural network configured to sequentially receive the plurality of multi-tags from the plurality of classifiers as a training data set, and as a result of training, generate a plurality of tags for the plurality of sentences based on the training data set. Each of the plurality of classifiers is configured to receive text fragments at different data points corresponding to a plurality of sentences having different context window sizes, and to generate multiple tags corresponding to the text fragments.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 693,959, filed on September 12, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments of this disclosure relate to the analysis of technical documentation for a storage device. Background Technology

[0004] The development of storage devices such as solid-state drives (SSDs) is a complex process, requiring expertise in stages such as integrated circuit design and verification, firmware development and testing, software simulation, and algorithm design. Most stages demand a deep understanding of various technical documents, such as specifications, datasheets, user guides, and product manuals. Therefore, the final product is based on a large number of features extracted from these technical documents. This activity can be automated using advanced natural language processing tools, saving engineers valuable time. Existing technical document analysis techniques are typically suitable for short, sequential, and fixed-volume texts (sentences, paragraphs) that are processed using binary classifiers. Embodiments of the present invention emerge in this context. Summary of the Invention

[0005] Various aspects of the present invention include systems and methods for analyzing technical documents of storage devices and extracting features from multiple sentences.

[0006] In one aspect of the invention, a system for analyzing at least one technical document on a storage device includes: a plurality of classifiers, each configured to receive a plurality of sentences from the technical document and generate multi-labels for the plurality of sentences, each of the multi-labels indicating whether each sentence possesses a target feature described in the technical document; and a combined neural network configured to sequentially receive the multi-labels from the plurality of classifiers as a training dataset, and, as a result of training, generate multiple labels for the plurality of sentences based on the training dataset. Each of the plurality of classifiers is configured to receive text fragments at different data points corresponding to the plurality of sentences with different context window sizes, and generate multi-labels corresponding to the text fragments.

[0007] In one aspect of the invention, a method for analyzing at least one technical document of a storage device includes: receiving a plurality of sentences from the technical document by each of a plurality of classifiers, and generating multi-labels for the plurality of sentences, each of the multi-labels indicating whether each sentence has a target feature described in the technical document; sequentially receiving the multiple multi-labels from the plurality of classifiers as a training dataset by a combined neural network; and generating multiple labels for the plurality of sentences based on the training dataset as a result of training the combined neural network. Receiving the plurality of sentences includes: receiving text fragments at different data points corresponding to the plurality of sentences with different context window sizes by each of the plurality of classifiers, and generating multi-labels corresponding to the text fragments.

[0008] Other aspects of the invention will become apparent from the following description. Attached Figure Description

[0009] Figure 1 This is a diagram illustrating a document analysis system and a system-on-chip (SoC) verification system according to an embodiment of the present invention.

[0010] Figure 2 This is a diagram illustrating a document analysis system with multiple multi-label classifiers according to an embodiment of the present invention.

[0011] Figure 3 A multi-label classifier according to an embodiment of the present invention is shown.

[0012] Figure 4 This is a diagram illustrating an example of a document analysis system with multiple multi-label classifiers according to an embodiment of the present invention.

[0013] Figure 5 This is a diagram of a neural network according to an embodiment of the present invention.

[0014] Figures 6A to 7B An example of extracting data from the M-PHY specification according to an embodiment of the present invention is shown.

[0015] Figure 8A and Figure 8B The performance of a trained classifier based on the context window size is shown according to an embodiment of the present invention.

[0016] Figure 9 This is a flowchart illustrating a document analysis method according to an embodiment of the present invention. Detailed Implementation

[0017] Various embodiments of the invention are described in more detail below with reference to the accompanying drawings. However, the invention may be implemented in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will fully convey the scope of the invention to those skilled in the art. Furthermore, references herein to “embodiment,” “another embodiment,” etc., do not necessarily refer to only one embodiment, and different references to any such phrase do not necessarily refer to the same embodiment. The term “embodiment” as used herein does not necessarily refer to all embodiments. Throughout this disclosure, the same reference numerals denote the same parts in the drawings and embodiments of the invention.

[0018] This invention can be embodied in many ways, including as a process; an apparatus; a system; a computer program product implemented on a computer-readable storage medium; and / or a processor, such as a processor adapted to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these embodiments or any other form in which the invention may take may be referred to as technology. Generally, the order of operations of the disclosed methods can be varied within the scope of this invention. Unless otherwise stated, components such as processors or memory described as suitable for performing tasks can be implemented as general components temporarily configured to perform tasks at a given time or manufactured as specific components for performing tasks. As used herein, the term "processor," etc., refers to one or more means, circuits, and / or processing cores suitable for processing data such as computer program instructions.

[0019] The methods, processes, and / or operations described herein can be executed by code or instructions to be run by a computer, processor, controller, or other signal processing device. The computer, processor, controller, or other signal processing device can be the computer, processor, controller, or other signal processing device described herein, or it can be an element other than those described herein. Because the algorithms underlying the method (or the operation of the computer, processor, controller, or other signal processing device) are described in detail, the code or instructions for implementing embodiments of the method can convert a computer, processor, controller, or other signal processing device into a dedicated processor for performing the methods herein.

[0020] When implemented at least in part as software, the controller, processor, device, module, unit, multiplexer, generator, logic circuit, interface, decoder, driver, and other signal generation and signal processing features may include, for example, a memory or other storage device for storing code or instructions to be executed by, for example, a computer, processor, microprocessor, controller, or other signal processing device.

[0021] The following appendices illustrate various aspects of the invention. Figure 1 The present invention provides a detailed description of embodiments thereof. The invention has been described in conjunction with such embodiments, but is not limited to any particular embodiment. The invention includes many alternatives, modifications, and equivalents. To provide a thorough understanding of the invention, numerous specific details are set forth in the following description. These details are provided for illustrative purposes; the invention may be practiced without some or all of these specific details. For clarity, technical materials known in the art related to the invention may not be described in detail.

[0022] Figure 1 This is a diagram illustrating a document analysis system 100 and a verification system 200 according to an embodiment of the present invention.

[0023] Reference Figure 1 The document analysis system 100 can analyze technical documents used to verify a designed system (e.g., a system-on-a-chip (SoC)). In various embodiments, the designed system can be an IP component of a storage device such as a NAND flash memory device, for example, a solid-state drive (SSD), an embedded multimedia card (eMMC), an open NAND flash interface (ONFi), universal flash memory (UFS), a low-power mobile industrial processor interface (MIPI) physical layer (M-PHY), high-speed non-volatile memory (NVMe), etc. In various embodiments, the technical documents may include at least one of specifications, datasheets, product manuals, and user guides.

[0024] Document analysis system 100 can provide analysis results to verification system 200. Verification system 200 can receive analysis results from document analysis system 100 and perform a verification process on the designed system based on the analysis results. Verification system 200 can verify whether the designed system meets the requirements described in the technical documentation of the designed system. The analysis results obtained from document analysis system 100 can also be used by verification engineers to design verification system 200.

[0025] Figure 2 This is a diagram illustrating a document analysis system according to an embodiment of the present invention.

[0026] Reference Figure 2 The document analysis system 100 can receive and analyze sentences from one or more large-capacity technical documents stored on a storage device. In one embodiment, the document analysis system 100 can detect and extract multiple sentence features from the large-capacity technical documents. The technical documents include at least one or more of specifications, manuals, user guides, and standards associated with the storage device.

[0027] Figure 2 The document analysis system 100 can perform a scheme to extract multi-sentence features from technical documents based on the following:

[0028] (1) Since the amount of text data being analyzed within a document is not fixed, multiple models (i.e., classifiers) with different context sizes (or context window sizes) S can be used to improve the quality of document analysis. Depending on the amount of available text data in the document, the parameter (i.e., the context size) S can vary from 2 sentences to 100 (or more) sentences.

[0029] (2) Since sentences within the volume of the analyzed text data can be connected in different ways to form features, i.e., sequentially or non-sequentially, a multi-label approach can be used for each model.

[0030] (3) The results of a single model with a fixed context window size S are worse in most cases than the combination of multiple models with different context window sizes S.

[0031] The document analysis system 100 may include multiple classifiers and a combined neural network 120. Figure 2 In the document analysis system 100 shown, multiple classifiers may include K multi-label classifiers 110_0 to 110_(K-1). Each of the multiple classifiers 110_0 to 110_(K-1) can receive multiple sentences from a technical document. Figure 2 In the document analysis system 100 shown, multiple models (i.e., K classifiers) with different context sizes S are configured. Classifier 110_0 receives sentences with a context size of S0, classifier 110_1 receives sentences with a context size of S1, and classifier 110_(K-1) receives sentences with a context size of S... K-1 The sentence. In one embodiment, a different context window can be determined as S0. <S1<……<S K-1 Each of the multiple classifiers 110_0 to 110_(K-1) can generate multi-labels for multiple sentences (e.g., classifier 110_0 generates label S0, classifier 110_1 generates label S1, classifier 110_(K-1) generates label S2, S3, S4, S5, S6, S7, S8, S9, S1, S1, S1, S1, S1, S1, S2, S1 ... K-1 Labels). Each label can indicate whether each sentence has the target feature (the desired feature). Each of the multiple classifiers 110_0 to 110_(K-1) can be based on a large language model (LLM).

[0032] Reference Figure 3The classifier 111 can receive data points (or text fragments) comprising multiple sentences (e.g., S sentences). The dataset comprising one or more data points can be based on one or more complete technical documents, and within these documents can be numerous text fragments with multiple connection sequences. Optionally, a text fragment can be a paragraph, page, or any other reasonable amount of text from one or more technical documents. The classifier 111 can classify the multiple sentences and generate multiple labels (e.g., S labels) for the multiple sentences based on the classification results. That is, one or more technical documents can be parsed into multiple sentences (e.g., S sentences) linked using labels (e.g., S labels).

[0033] In one embodiment, the label value can be a binary value indicating whether the corresponding sentence has the desired feature described in the document. For example, a label value (1) can indicate that the corresponding sentence has the desired feature, and a label value (0) can indicate that the corresponding sentence does not have the desired feature. In another embodiment, the range of label values ​​can include more than two values ​​(e.g., the desired feature has low (1), medium (2), or high (3) values). Optionally, the label value can be a non-integer probability value (in the range of 1 to 0) of having the feature.

[0034] Reference Figure 4 Multiple models can be trained on the same dataset, but divided into different numbers of data points. If the dataset has N sentences and the window size is S, the number of data points for the classifier is determined by the top function, hereinafter referred to as... ceil ( N / S Therefore, a larger context window size S is chosen, and a smaller dataset is used for the training process. Since different documents require different context window sizes, multiple (i.e., K) multi-label classifiers can be used to form a combined model. The top function used in this paper is a mathematical function that rounds a real number up to the smallest integer greater than or equal to that real number.

[0035] exist Figure 4 In the above, the number of data points for classifiers 110_0, 110_1, ..., 110_(K-1) can be ceil(N / S0), ceil(N / S1), ..., ceil(N / S) respectively. K-1 In other words, classifier 110_0 receives data points (text fragments) corresponding to the number of ceil(N / S0). Text fragment Text0 (S_0) (i.e., datapoint0) (S_0) (This can include multiple sentences, from Sentence0 to Sentence...) (S0-1) Text fragmentN / S_0 (S_0) (i.e., datapoint) N / S_0 (S_0) The classifier can include multiple sentences from Sentence_floor(N / S0)×S0 to Sentence_ceil(N / S0)×S0. Classifier 110_(K-1) receives the values ​​of ceil(N / S0)×S0. K-1 The number of data points (text fragments) corresponding to the number of (text fragments). Text fragment Text0 (S_K-1) (i.e., datapoint0) (S_K-1) (This can include multiple sentences, from Sentence 0 to Sentence 1) (SK-1-1) Text fragment N / S_(K-1) (S_K-1) (i.e., datapoint) N / S_(K-1) (S_K-1) It can include multiple sentences. Sentence_floor (N / S) (K-1) )×S (K-1) To Sentence_ceil (N / S (K-1) )×S (K-1) The base function used in this article is a mathematical function that rounds a real number down to the largest integer less than or equal to that real number.

[0036] Each model (i.e., the classifier) ​​can generate N labels as the training dataset for the combined neural network 120: for the context window size S 0, generate Label0 (S_0) Label N-1 (S_0) Regarding the context window size S 1. Generate Label0 (S_1) Label N-1 (S_1) And regarding the context window size S K-1 Generate Label0 (S_(K-1)) Label N-1 (S_(K-1)) .

[0037] have S i The classifier receives the sentence context window. ceil ( N / S i ) data points, each of which includes S i One sentence. If N Cannot be Si If divisible, the final data point can be appended with some text {N-( floor ( N / S i ) × S i )} sentences, so that the number of sentences in the final data points is exactly S i .

[0038] In one embodiment, the plurality of classifiers includes: a first classifier configured to receive a first number of text fragments based on the number of labels N and a first context size; and a second classifier configured to receive a second number of text fragments based on the number of labels N and a second context size, the second context size being different from the first context size. In one embodiment, each of the first and second context sizes is variable. In one embodiment, the first number of text fragments is determined based on a top function between the number of multi-labels and the first context size, and the second number of text fragments is determined based on a top function between the number of multi-labels and the second context size.

[0039] All ( N × K) The labels can form a dataset for training the ensemble neural network 120, which adjusts the weights to perform training based on the data provided by the classifier. N × K) The neural network 120 uses K labels to predict the final K label values ​​for each sentence. Therefore, the combined neural network 120 can provide a balanced prediction of the desired features for the document text, taking into account context windows of different sizes.

[0040] Reference Figure 2 and Figure 4 Each of the multiple classifiers 110_0 to 110_(K-1) can receive text fragments at different data points corresponding to multiple sentences with different context window sizes, and generate multi-labels corresponding to the text fragments. The ensemble neural network 120 can sequentially receive multiple multi-labels as a training dataset from the multiple classifiers 110_0 to 110_(K-1), and as a training result, generate multiple labels for multiple sentences based on the training dataset. For example, as a training result, the ensemble neural network 120 can generate N labels. (E) Label N-1 (E) The combined neural network 120 can be used based on different context windows. S 0< S 1<……<S K-1 A model is a set of K models. A combined model can represent a machine learning technique that combines multiple models (i.e., multiple classifiers 110_0 to 110_(K-1)) to improve prediction accuracy.

[0041] In this way, the document analysis system 100 can detect and extract multi-sentence features from a large volume of technical documents. In one embodiment, sequential text fragments within a specific page of the technical document (e.g., 10 sentences (sentences 1 to 10)) can be detected. In another embodiment, non-sequential text fragments (e.g., sentences 1, 3, 7, 8, 9, and 10) can be detected if they are classified as the desired target features described in the technical document. Therefore, the scheme of the document analysis system 100 is optimized based on the results obtained from analyzing a fixed number of sentences S from K multi-label classifiers.

[0042] Figure 5 This is a diagram of a neural network 1100 according to an embodiment of the present invention. The neural network 1100 can be implemented as a plurality of classifiers 110_0 to 110_(K-1), each classifier being configured to classify text segments and generate multi-labels based on a large language model (LLM). Furthermore, the neural network 1100 can be implemented as a combined neural network 120.

[0043] Reference Figure 5 A feature map 1102 associated with one or more input conditions can be input into a neural network 1100. The feature map 1102 includes one or more features associated with one or more input conditions. The neural network 1100 uses the feature map 1102 to generate and output information 1104. As shown, the neural network 1100 includes an input layer 1110, one or more hidden layers 1120, and an output layer 1130. Features from the feature map 1102 can be connected to input nodes in the input layer 1110. Information 1104 can be generated based on the output nodes of the output layer 1130. One or more hidden layers 1120 can exist between the input layer 1110 and the output layer 1130. The neural network 1100 can be pre-trained to process features from the feature map 1102 through different layers 1110, 1120, and 1130 to output information 1104.

[0044] Neural network 1100 can be a multi-layer neural network, which represents a network of interconnected nodes, such as an artificial deep neural network. In such a network, knowledge about nodes (e.g., information about specific features represented by nodes) is shared between layers, while knowledge specific to each layer is also preserved. Each node represents a piece of information. Knowledge can be exchanged between nodes through node-to-node interconnections. Input to neural network 1100 can activate a set of nodes. Subsequently, this set of nodes can activate other nodes, thereby propagating knowledge about the input. This activation process can be repeated on other nodes until nodes in output layer 1130 are selected and activated.

[0045] In one embodiment, the neural network 1100 may include a hierarchy of layers, which represents a hierarchy of nodes interconnected in a feedforward manner. The input layer 1110 may exist at the lowest level. As detailed below, the input layer 1110 may include a set of nodes, referred to herein as input nodes (e.g., ...). Figure 4 (The training dataset). When feature map 1102 is input into neural network 1100, each input node in input layer 1110 can be connected to each feature of feature map 1102. Each connection can have weights, each of which is derived from the training of neural network 1100. Weights represent a set of parameters of neural network 1100. Input nodes can transform features by applying activation functions to them. Information obtained from the transformations can be passed to higher-level nodes.

[0046] Output layer 1130 may exist at the highest level. Output layer 1130 may include one or more output nodes. When output layer 1130 outputs output information 1104, each output node can provide a specific value of the output information 1104 (e.g., the N labels of the combined neural network 120 obtained as a training result). (E) Label N-1 (E) The number of output nodes depends on the required number of specific values ​​for output information 1104. In other words, there can be a one-to-one relationship or mapping between the number of output nodes and the number of values ​​or lines of output information 1104.

[0047] Hidden layer 1120 may exist between input layer 1110 and output layer 1130. There may be L hidden layers 1120, where "L" is an integer greater than or equal to 1. Each of the hidden layers 1120 may include a group of nodes, referred to herein as a hidden node. Exemplary hidden layers may include upsampling layers, convolutional layers, fully connected layers, and / or data transformation layers.

[0048] In the lowest layer of hidden layer 1120, the hidden nodes of this layer can be interconnected with the input nodes. In the highest layer of hidden layer 1120, the hidden nodes of this layer can be interconnected with the output nodes. Input nodes may not be directly interconnected with output nodes. If multiple hidden layers exist, the input nodes are interconnected with the hidden nodes of the lowest hidden layer. Subsequently, these hidden nodes are interconnected with the hidden nodes of the next hidden layer. Interconnections can represent a learned piece of information about the two interconnected nodes. Interconnections can have numerical weights that can be adjusted (e.g., adjusted based on the training dataset) so that the neural network 1100 adapts to the input and is able to learn.

[0049] Typically, hidden layer 1120 allows knowledge about the input nodes of input layer 1110 to be shared among the output nodes of output layer 1130. For this purpose, the transformation can be performed through hidden layer 1120. Applied to the input node. In the example, the transformation... It is non-linear. Different non-linear transformations It is available, including, for example, rectifier functions. In the example, cross-validation is used to select a specific nonlinear transformation. .

[0050] Example

[0051] Figure 4 The training dataset is based on the M-PHY 4.1 specification (the M-PHY 4.1 specification is a specification for physical layer interfaces). The target feature analyzed by the document analysis system 100 is to predict whether a given text fragment is a requirement in the M-PHY 4.1 specification. For verification purposes, the specification has been manually analyzed by extracting requirements and designing a test environment to check the correctness of protocol operations. Therefore, the specification has been divided into 219 pages of text, including 4629 sentences: 772 sentences are requirement-related and 3857 sentences are not requirement-related.

[0052] Figures 6A to 7B This shows an example of data extracted from page 24 of the M-PHY Specification. Figure 6A and Figure 6B Page 24 of the M-PHY specification is shown, and Figure 7A and Figure 7B This shows data extracted from page 24 of the M-PHY specification. Figure 7A and Figure 7B In the text, 610 represents the extracted sentence, and 620 represents the tag generated by the document analysis system.

[0053] The fundamental element of the ensemble model (i.e., ensemble neural network 120) is an LLM-based multi-label classifier. In one embodiment, it has been...S The -label Mistral v.0.1 model is used as the base classifier. A classifier (43 classifiers) with different parameters S = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100} has been trained on the dataset (70% of the dataset is training data, and 30% is validation data). Figure 8A The diagram shows the performance of a trained classifier based on the context window size S. That is, in... Figure 8A In the diagram, the x-axis represents the context window size, and the y-axis represents the performance of the trained classifier (i.e., the classifier's F1-score). Due to the reduced number of actual data points, a larger context window size results in a lower F1-score (indicating poorer performance). Typically, an F1-score greater than 0.5 indicates that, in the case of binary classification, the model performs better than a random guessing algorithm.

[0054] Because classifiers have validation sets of varying sizes (within sentences), it is difficult to evaluate them using the same metric. Therefore, the number of sentences misclassified throughout the entire specification was chosen as the metric to compare the performance of a single classifier with that of a combined model utilizing one or more classifiers. Because the number of classifiers is relatively large (43), it is impossible to evaluate all possible combinations (2). 43 Instead, the classifiers are combined sequentially: first, the combined model includes only one classifier (S={2}), second, the combined model includes two classifiers (S={2, 3}), third, the combined model includes three classifiers (S={2, 3, 4}), ... and the 43rd combined model includes 43 classifiers (S={2, 3, ..., 30, 35, 40, ..., 100}).

[0055] Figure 8B The diagram shows a comparison between the number of errors (o) of 43 individual classifiers and the number of errors (□) of 43 combined models. Figure 8B In the diagram, the x-axis represents the context window size, and the y-axis represents the number of errors for a single classifier and the combined model. The combined model makes significantly fewer errors compared to a single classifier, especially when the number of models with different context windows in the combined model is greater than 5–10.

[0056] Re-reference Figure 4 and Figure 5 The architecture of the combined neural network 120 can be implemented using, for example, a 5-layer fully connected network comprising one input layer, four hidden layers, and one output layer (e.g., K linear neurons (input layer), 1024 ReLU neurons (hidden layers), 512 ReLU neurons (hidden layers), 256 ReLU neurons (hidden layers), 128 ReLU neurons (hidden layers), and 1 Sigmoid neuron (output layer)). ReLU represents the rectified linear unit activation function.

[0057] To achieve a negligible number of errors, it is sufficient to include at least K=11 classifiers in the ensemble model, resulting in only 8 errors. On the other hand, using K=16 classifiers in the ensemble model provides even better results, i.e., 0 errors. Therefore, increasing the number of classifiers from K=11 to K=16 is optimal in terms of both quality (number of errors) and size (number of classifiers used in the ensemble model). Increasing the number of classifiers in the ensemble model does not significantly improve performance; that is, some combinations may produce 1-2 errors, but the overall quality remains comparable.

[0058] Unfortunately, the amount of data from a single specification is insufficient to provide robust performance on unknown data. Inference using a combined model with K=16 classifiers has been tested on the M-PHY specification v.6.0. The combined model recognized 100% of the requirements inherited from the M-PHY v. 4.1 specification, but only 70% of the new requirements. These results demonstrate an overfitting problem, which can be addressed by using more labeled data extracted from various technical documents for the training dataset.

[0059] Figure 9 This is a flowchart illustrating a document analysis method 900 according to an embodiment of the present invention. Method 900 can be derived from... Figures 2 to 4 The document analysis system executes to analyze documents to be used for verification of storage devices.

[0060] Reference Figure 9 In operation 910, method 900 may include: receiving multiple sentences from a technical document by each of a plurality of classifiers, and generating multi-labels for the multiple sentences. Each of the multi-labels may indicate whether each sentence has the target features described in the technical document.

[0061] Operation 920 may include: receiving multiple multi-labels as training datasets sequentially from multiple classifiers by a combined neural network.

[0062] Operation 930 may include: as a result of training, generating multiple labels for multiple sentences by a combined neural network based on the training dataset.

[0063] Receiving multiple sentences can include: each of multiple classifiers receiving text fragments at different data points corresponding to multiple sentences with different context window sizes, and generating multi-labels corresponding to the text fragments.

[0064] In one embodiment, receiving multiple sentences includes: receiving a first number of text fragments by a first classifier based on the number of multi-labels and a first context size, and receiving a second number of text fragments by a second classifier based on the number of multi-labels and a second context size, wherein the second context size is different from the first context size.

[0065] In one embodiment, each of the first context size and the second context size is variable.

[0066] In one embodiment, the method further includes: determining a first number of text fragments based on a top function between the number of multi-labels and a first context size, and determining a second number of text fragments based on a top function between the number of multi-labels and a second context size.

[0067] In one embodiment, each of the multiple classifiers classifies a text segment and generates multi-labels based on a large language model (LLM).

[0068] In one embodiment, each of the multiple labels includes one of two binary values ​​for the target feature.

[0069] In one embodiment, each of the multiple tags includes a value within a range of values ​​having more than two binary values ​​for the target feature.

[0070] In one embodiment, each of the multiple labels includes a probability value for the target feature.

[0071] In one embodiment, the combined neural network includes a 5-layer connection network, which includes an input layer, four hidden layers, and an output layer.

[0072] In one embodiment, the technical documentation includes at least one or more of the respective specifications, manuals, user guides, and standards associated with the storage device.

[0073] As described above, embodiments of the present invention provide a scheme for analyzing technical documents on a storage device and extracting multi-sentence features from the technical documents based on a combined learning technique with multiple multi-label classifiers. This scheme can be used for relatively large amounts of text and provides input for engineers (verification, firmware, software) to save valuable time for handling higher-priority tasks.

[0074] Although the embodiments described above have been shown and described in detail for clarity and understanding, the invention is not limited to the details provided. Those skilled in the art will understand from the above disclosure that many alternatives exist for implementing the invention. Therefore, the disclosed embodiments are illustrative and not restrictive. The invention is intended to encompass all modifications and substitutions. Furthermore, these embodiments can be combined to form other embodiments.

Claims

1. A system for analyzing at least one technical document of a storage device, the system comprising: Multiple classifiers, each receiving multiple sentences from the technical document and generating multiple labels for the multiple sentences, each of the multiple labels indicating whether each sentence has the target features described in the technical document; as well as A combined neural network sequentially receives multiple multi-labels as training datasets from the multiple classifiers, and generates multiple labels for the multiple sentences as training results based on the training datasets. Each of the plurality of classifiers receives text fragments at different data points corresponding to multiple sentences with different context window sizes, and generates the multi-labels corresponding to the text fragments.

2. The system according to claim 1, wherein, The multiple classifiers include: A first classifier receives a first number of text segments based on the number of multi-labels and a first context size, and The second classifier receives a second number of text fragments based on the number of multi-labels and a second context size, where the second context size is different from the first context size.

3. The system according to claim 2, wherein, Each of the first context size and the second context size is variable.

4. The system according to claim 1, wherein, Each of the multiple classifiers classifies the text segment and generates the multi-label based on a large language model, i.e., LLM.

5. The system according to claim 1, wherein, Each of the multiple tags includes one of two binary values ​​for the target feature.

6. The system according to claim 1, wherein, Each of the multiple tags includes a value within a range of values ​​having more than two binary values ​​for the target feature.

7. The system according to claim 1, wherein, Each of the multiple labels includes a probability value for the target feature.

8. The system according to claim 1, wherein, The combined neural network includes a connection network, which includes an input layer, four hidden layers, and an output layer.

9. The system according to claim 1, wherein, The technical documentation includes at least one or more of the specifications, manuals, user guides and standards associated with the storage device.

10. A method for analyzing at least one technical document of a storage device, the method comprising: Each of a plurality of classifiers receives a plurality of sentences from the technical document and generates a plurality of labels for the plurality of sentences, each of the plurality of labels indicating whether each sentence has the target features described in the technical document; The combined neural network sequentially receives multiple multi-label data as training datasets from the multiple classifiers; as well as The combined neural network generates multiple labels for the multiple sentences based on the training dataset as the training result. The receiving of multiple sentences includes: Each of the plurality of classifiers receives text fragments at different data points corresponding to multiple sentences with different context window sizes, and Generate the multi-tags corresponding to the text fragment.

11. The method according to claim 10, wherein, The receipt of multiple sentences includes: The first classifier receives a first number of text fragments based on the number of multi-labels and the first context size, and A second classifier receives a second number of text fragments based on the number of multi-labels and a second context size, where the second context size is different from the first context size.

12. The method according to claim 11, wherein, Each of the first context size and the second context size is variable.

13. The method according to claim 10, wherein, Each of the multiple classifiers classifies the text segment and generates the multi-label based on a large language model, i.e., LLM.

14. The method of claim 10, wherein, Each of the multiple tags includes one of two binary values ​​for the target feature.

15. The method according to claim 10, wherein, Each of the multiple tags includes a value within a range of values ​​having more than two binary values ​​for the target feature.

16. The method of claim 10, wherein, Each of the multiple labels includes a probability value for the target feature.

17. The method according to claim 10, wherein, The combined neural network includes a connection network, which includes an input layer, four hidden layers, and an output layer.

18. The method according to claim 10, wherein, The technical documentation includes at least one or more of the specifications, manuals, user guides and standards associated with the storage device.

Images