Analysis method, system and equipment based on natural language processing and deep learning and medium

Abstract

The invention relates to the technical field of software demand analysis, in particular to an analysis method, system and equipment based on natural language processing and deep learning and a medium. The method comprises the following steps: firstly, preprocessing acquired software demand text data to obtain a software demand data set and tagging part-of-speech; then, according to the marked software demand data set, constructing a demand classification and view recognition model, and extracting to obtain demand text features; and finally, constructing a software demand knowledge graph according to the demand text features, calling a visualization library, and visualizing the demand classification and view recognition model in a graphical form. Through deep mining of massive software demand text data, accurate identification of demand contents and types and sorting of logic relations among demands, an efficient and accurate demand analysis tool is provided for software developers, demand analysts and the like, the demand analysis period is greatly shortened, the labor cost is reduced, and the software product quality is improved.

Description

Technical Field

[0001] This invention relates to the field of software analysis technology, and more specifically, to an analysis method, system, device, and medium based on natural language processing and deep learning. Background Technology

[0002] In today's digital age, software systems are becoming increasingly complex, and software demands are growing exponentially, making traditional requirements analysis and modeling methods inadequate. Statistics show that over 70% of software projects suffer from cost overruns and delays due to misunderstandings of requirements and poor communication. With the rapid development of artificial intelligence, Natural Language Processing (NLP) and deep learning offer new opportunities for software requirements analysis. NLP enables computers to understand human language, transforming unstructured requirements documents into machine-processable formats; deep learning's powerful feature learning capabilities can uncover potential relationships between requirements, enabling intelligent classification and prediction. Currently, while some related research exists, there are still shortcomings in requirements completeness, accuracy analysis, and complex scenario modeling. This project aims to fill these gaps, improve the success rate of software projects, and drive industry development. Summary of the Invention

[0003] This invention addresses the shortcomings of existing analysis methods in terms of completeness, accuracy, and modeling of complex scenarios. It proposes an analysis method, system, device, and medium based on natural language processing and deep learning. Through in-depth mining of massive amounts of software requirement text data, it accurately identifies requirement content and types, clarifies the logical relationships between requirements, and automatically generates a visual requirement model. The system will possess functions such as requirement classification, conflict detection, and traceability analysis, providing software developers and requirement analysts with efficient and accurate requirement analysis tools, significantly shortening the requirement analysis cycle, reducing labor costs, and improving software product quality.

[0004] The specific implementation details of this invention are as follows: An analysis method based on natural language processing and deep learning specifically includes the following steps: Step S1: Preprocess the acquired software requirement text data to obtain a software requirement dataset and tag it with parts of speech; Step S2: Based on the labeled software requirement dataset, construct a requirement classification and view recognition model to extract requirement text features; Step S3: Construct a software requirement knowledge graph based on the characteristics of the requirement text, and call the visualization library to visualize the requirement classification and view recognition model in a graphical form.

[0005] To better realize the present invention, step S1 further includes the following steps: Step S11: Call the NLP toolkit to preprocess the obtained software requirement text data; Step S12: After word segmentation and preprocessing, the software requirement text data is labeled with part-of-speech tags to obtain the labeled software requirement dataset; Step S13: Convert unstructured requirement data into structured format and store it according to the set text parsing rules.

[0006] To better realize the present invention, step S12 further includes the following steps: Step S121: Based on the spaCy natural language processing library, modify the Jieba word segmenter rules and construct the software requirement lexicon; Step S122: Traverse the preprocessed software requirement text data and call the pipeline segmentation of the NLP toolkit to traverse the software requirement text data; Step S123: Call token.pos_ to obtain part-of-speech tags from the segmented software requirement text data and annotate them; Step S124: Assemble the labeled software requirement data into a structured dictionary, and combine the dictionary into a list to obtain the software requirement dataset.

[0007] To better realize the present invention, step S2 further includes the following steps: Step S21: Based on the labeled software requirement dataset, call the neural network to build a requirement classification and view recognition model; Step S22: Based on the labeled software requirement data, label the requirement categories and intent tags, train the requirement classification and view recognition model, and extract the requirement text features; Step S23: Calculate the difference between the predicted value and the true label by calling the cross-entropy loss function, and update the model parameters by calling the backpropagation algorithm.

[0008] To better realize the present invention, step S22 further includes the following steps: Step S221: Use the TfidfVectorizer tool to convert the labeled text into a feature matrix; Step S222: Construct a vocabulary, map the annotated requirement text to integer indices, and convert the integer sequence into a dense word vector sequence in the Embedding layer to obtain integer labels for requirement categories and integer labels for intent labels.

[0009] To better realize the present invention, step S3 further includes the following steps: Step S31: Construct a software requirement knowledge graph by using the features of the requirement text as nodes and the relationships as edges; Step S32: Invoke the graph neural network to mine the implicit relationships in the constructed software requirements knowledge graph; Step S33: Call the visualization library to read the requirement knowledge graph data from the graph database and visualize the requirement classification and view recognition model in a graphical form.

[0010] To better realize the present invention, step S33 further includes the following steps: Step S331: Obtain the subgraph to be visualized based on the constructed Cypher query; Step S332: Merge the implicit relationships of the acquired software requirement knowledge graph as edge data with the software requirement knowledge graph to obtain the integrated model; Step S333: Build the Flask backend API. Based on the AJAX request data obtained from the frontend, call the visualization library to render the integrated model and generate interactive charts.

[0011] Based on the analysis method based on natural language processing and deep learning proposed above, in order to better realize the present invention, a further analysis system based on natural language processing and deep learning is proposed to execute the above-mentioned analysis method based on natural language processing and deep learning; including a preprocessing unit, a demand classification and intent recognition unit, and a visualization unit; The preprocessing unit is used to preprocess the acquired software requirement text data to obtain a software requirement dataset and tag it with part-of-speech tags. The requirement classification and intent recognition unit is used to construct a requirement classification and view recognition model based on the labeled software requirement dataset and extract requirement text features. The visualization unit is used to construct a software requirement knowledge graph based on the characteristics of the requirement text, call the visualization library, and visualize the requirement classification and view recognition model in a graphical form.

[0012] Based on the analysis method based on natural language processing and deep learning proposed above, in order to better realize the present invention, an electronic device is further proposed, including a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, the above-mentioned analysis method based on natural language processing and deep learning is implemented.

[0013] Based on the analysis method based on natural language processing and deep learning proposed above, in order to better realize the present invention, a computer-readable storage medium is further proposed, wherein computer instructions are stored on the computer-readable storage medium; when the computer instructions are executed on the above-mentioned electronic device, the above-mentioned analysis method based on natural language processing and deep learning is realized.

[0014] The present invention has the following beneficial effects: (1) This invention accurately identifies the content and type of requirements by deeply mining massive amounts of software requirement text data, sorts out the logical relationships between requirements, and provides software developers, requirement analysts and others with efficient and accurate requirement analysis tools, which greatly shortens the requirement analysis cycle, reduces labor costs and improves the quality of software products.

[0015] (2) This invention improves the accuracy of word segmentation by segmenting words and tagging parts of speech, laying the foundation for building a high-quality demand analysis model and ensuring the reliability and availability of input data.

[0016] (3) This invention uses graph neural networks to mine hidden relationships in graphs and realizes automatic reasoning of demand relationships. Attached Figure Description

[0017] Figure 1 A schematic flowchart of the analysis method based on natural language processing and deep learning provided by the present invention. Detailed Implementation

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be regarded as a limitation on the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0020] Example 1: This embodiment proposes an analysis method based on natural language processing and deep learning, which specifically includes the following steps: Step S1: Preprocess the acquired software requirement text data to obtain a software requirement dataset and tag it with parts of speech; Step S1 specifically includes the following steps: Step S11: Call the NLP toolkit to preprocess the obtained software requirement text data; Step S12: After word segmentation and preprocessing, the software requirement text data is labeled with part-of-speech tags to obtain the labeled software requirement dataset; Step S12 specifically includes the following steps: Step S121: Based on the spaCy natural language processing library, modify the Jieba word segmenter rules and construct the software requirement lexicon; Step S122: Traverse the preprocessed software requirement text data and call the pipeline segmentation of the NLP toolkit to traverse the software requirement text data; Step S123: Call token.pos_ to obtain part-of-speech tags from the segmented software requirement text data and annotate them; Step S124: Assemble the labeled software requirement data into a structured dictionary, and combine the dictionary into a list to obtain the software requirement dataset.

[0021] Step S13: Convert unstructured requirement data into structured format and store it according to the set text parsing rules.

[0022] This embodiment first performs preprocessing of the software requirements text. Software requirements come from diverse sources, with varying formats and quality, and contain a significant amount of noisy information. This embodiment will explore in depth preprocessing methods for requirements text, including cleaning, noise reduction, word segmentation, and part-of-speech tagging. For common domain-specific vocabulary in the requirements text, a professional thesaurus is developed to assist in word segmentation, improving segmentation accuracy. A combination of rules and statistics is used to identify and correct spelling errors in the text. Simultaneously, it investigates how to transform unstructured text into structured data to facilitate subsequent model processing, laying the foundation for building a high-quality requirements analysis model and ensuring the reliability and usability of the input data.

[0023] Step S2: Based on the labeled software requirement dataset, construct a requirement classification and view recognition model to extract requirement text features; Step S2 specifically includes the following steps: Step S21: Based on the labeled software requirement dataset, call the neural network to build a requirement classification and view recognition model; Step S22: Based on the labeled software requirement data, label the requirement categories and intent tags, train the requirement classification and view recognition model, and extract the requirement text features; Step S23: Calculate the difference between the predicted value and the true label by calling the cross-entropy loss function, and update the model parameters by calling the backpropagation algorithm.

[0024] Step S23 specifically includes the following steps: Step S221: Call the train_test_split function to divide the labeled software requirements dataset into training set, validation set, and test set; Step S222: Use the TfidfVectorizer tool to convert the labeled text into a feature matrix; Step S223: Construct a vocabulary, map the annotated requirement text to integer indices, and convert the integer sequence into a dense word vector sequence in the Embedding layer to obtain integer labels for requirement categories and integer labels for intent labels.

[0025] Step S3: Construct a software requirement knowledge graph based on the characteristics of the requirement text, and call the visualization library to visualize the requirement classification and view recognition model in a graphical form.

[0026] This embodiment then performs deep learning-based requirement classification and intent recognition, which are core components of requirement analysis. A requirement classification and intent recognition model is constructed based on deep learning models such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variants LSTM and GRU. Samples of software requirements from multiple domains and types are collected, and requirement categories and intent labels are annotated to train the model to learn requirement features. By optimizing the network structure and tuning hyperparameters, the model's accuracy and recall in requirement classification and intent recognition tasks are improved. For novel and complex requirement scenarios, an attention mechanism is introduced to enable the model to focus on key information, improve its generalization ability, and accurately determine the category of the requirement and the user's intent.

[0027] Step S3 specifically includes the following steps: Step S31: Construct a software requirement knowledge graph by using the features of the requirement text as nodes and the relationships as edges; Step S32: Invoke the graph neural network to mine the implicit relationships in the constructed software requirements knowledge graph; Step S33: Call the visualization library to read the requirement knowledge graph data from the graph database and visualize the requirement classification and view recognition model in a graphical form.

[0028] Step S33 specifically includes the following steps: Step S331: Obtain the subgraph to be visualized based on the constructed Cypher query; Step S332: Merge the implicit relationships of the acquired software requirement knowledge graph as edge data with the software requirement knowledge graph to obtain the integrated model; Step S333: Build the Flask backend API. Based on the AJAX request data obtained from the frontend, call the visualization library to render the integrated model and generate interactive charts.

[0029] This embodiment concludes with software requirement relationship modeling and visualization. Software requirements exhibit various complex relationships, such as dependency, conflict, and inclusion. This embodiment utilizes knowledge graph technology, treating requirements as nodes and relationships as edges to construct a requirement knowledge graph, visually displaying the structure between requirements. Graph Neural Networks (GNNs) are used to mine implicit relationships within the graph, enabling automatic reasoning about requirement relationships. Simultaneously, requirement model visualization technology is researched, developing a simple and intuitive visualization interface that allows users to graphically browse and edit requirement models, facilitating requirement communication and verification, assisting project teams in fully understanding the requirement architecture, and promptly identifying potential problems.

[0030] Working principle: This embodiment deeply mines massive amounts of software requirement text data to accurately identify the content and type of requirements, clarify the logical relationships between requirements, and automatically generate a visual requirement model. The system will have functions such as requirement classification, conflict detection, and traceability analysis, providing software developers and requirement analysts with efficient and accurate requirement analysis tools, significantly shortening the requirement analysis cycle, reducing labor costs, and improving software product quality.

[0031] Example 2: This embodiment is based on the above embodiment 1, such as... Figure 1 As shown, a specific embodiment will be described in detail, which includes the following steps.

[0032] Step S1: Preprocess the acquired software requirement text data to obtain a software requirement dataset and tag it with parts of speech; Step S1 specifically includes the following steps: Step S11: Call the NLP toolkit to preprocess the obtained software requirement text data; The software requirement text obtained includes source files (such as .txt, .docx, .pdf, .xlsx) or text fields in a database that store the software requirement text; write file reading functions, using the appropriate libraries (such as python-docx for Word, PyPDF2 for PDF, pandas for Excel) according to different file formats; extract plain text content from files or databases and store it in Python string variables or string lists; Next, the text is cleaned and normalized. Regular expressions (re library) are used to remove or replace special characters in the text (such as HTML tags / <[^>]*> / , redundant tabs \t, newlines \n, etc.), but semantically important periods, commas, etc. are retained; irregular spaces (such as multiple consecutive spaces) are replaced with single spaces; the entire text is converted to lowercase to ensure the consistency of model processing (for example, "User" and "user" are treated as the same word); the text encoding is ensured to be consistent (such as UTF-8), garbled characters are processed, and a cleaned long text string is obtained. Then, based on the cleaned long text string, use nltk.sent_tokenize() or the sentence segmentation function of the spaCy model to divide the large section of required text into independent sentences. Subsequent tokenization and part-of-speech tagging are usually performed at the sentence level, which can improve processing accuracy and interpretability, resulting in a list of sentences. Finally, initialize the selected NLP toolkit in the code. For spaCy: nlp = spacy.load("en_core_web_sm"); for NLTK: ensure that the necessary resource packages (such as punkt, averaged_perceptron_tagger) have been downloaded.

[0033] Step S12: After word segmentation and preprocessing, the software requirement text data is labeled with part-of-speech tags to obtain the labeled software requirement dataset; Step S12 specifically includes the following steps: Step S121: Based on the spaCy natural language processing library, modify the Jieba word segmenter rules and construct the software requirement lexicon; Step S122: Traverse the preprocessed software requirement text data and call the pipeline segmentation of the NLP toolkit to traverse the software requirement text data; Step S123: Call token.pos_ to obtain part-of-speech tags from the segmented software requirement text data and annotate them; Step S124: Assemble the labeled software requirement data into a structured dictionary, and combine the dictionary into a list to obtain the software requirement dataset.

[0034] This embodiment decomposes each sentence into meaningful lexical units (word segmentation) and labels each lexical unit with its part of speech, forming structured data that can be used for subsequent analysis.

[0035] First, integrate the custom thesaurus and load the custom thesaurus file (custom_lexicon.txt) prepared in the "Environment Setup" phase. For Jieba: jieba.load_userdict("path / to / custom_lexicon.txt"), for spaCy: ensure that domain terms (such as "Single Sign-On", "QoS") are not segmented by modifying the segmenter rules or creating entity matching rules. Next, word segmentation and part-of-speech tagging are performed. Each sentence is traversed according to the sentence list, and the processing pipeline of the NLP toolkit is invoked. The specific process is as follows: doc = nlp(sentence) tokens = [token.text for token in doc] pos_tags = [token.pos_ for token in doc] detailed_pos_tags = [token.tag_ for token in doc]; Then, handle word segmentation ambiguity and out-of-vocabulary words, and check the word segmentation results, especially for words and domain abbreviations in the custom thesaurus, to see if they are correctly segmented; for words that are not processed correctly, they can be added to the custom thesaurus and reloaded, or post-processing rules can be written to force merging; Finally, the labeled dataset is assembled, and the word segmentation results, part-of-speech tagging results, and original sentence processing results of all sentences are organized into a structured dictionary. Then, all the sentences in the dictionary are combined into a list. This is the "labeled software requirements dataset".

[0036] Step S13: Convert unstructured requirement data into structured format and store it according to the set text parsing rules.

[0037] In this embodiment, the annotated_dataset is saved as a JSON or CSV file so that it can be directly loaded and used in subsequent steps without reprocessing.

[0038] Step S2: Based on the labeled software requirement dataset, construct a requirement classification and view recognition model to extract requirement text features; Step S2 specifically includes the following steps: Step S21: Based on the labeled software requirement dataset, call the neural network to build a requirement classification and view recognition model; This embodiment first loads the labeled data, reads the labeled dataset processed in the previous steps from the JSON / CSV file, and extracts the text data (processed_tokens or original_sentence) and the corresponding labels (such as classification categories: functional requirements, performance requirements, security requirements, etc.).

[0039] Next, text vectorization is performed. Use sklearn.feature_extraction.text.CountVectorizer or TfidfVectorizer to convert the text into bag-of-words model vectors; or use pre-trained word embeddings (such as Word2Vec, GloVe) or contextual embeddings (BERT output) to convert each word into a dense vector, and then represent the sentence as a sequence or aggregation (such as summation, averaging) of word vectors.

[0040] Finally, the dataset was split using sklearn.model_selection.train_test_split. First, it was divided into a training set (70%) and a temporary test set (30%). Then, the temporary test set was divided into a validation set (15%) and a test set (15%). The final ratio was: training set: validation set: test set = 70:15:15. This ensured that the class distribution of each set was consistent with the overall distribution.

[0041] When using a neural network to build a demand classification and view recognition model, the input layer is defined first, with its shape (input_dim: vocabulary size, input_length: sequence length) defined according to the text vectorization method. Next, an embedding layer is defined, adding an embedding layer to map integer indices to dense vectors; this can be randomly initialized or pre-trained embeddings can be loaded. Then, a feature extractor is built, adding a one-dimensional convolutional layer (Conv1D) to capture local n-gram features, a global max pooling layer (GlobalMaxPooling1D) or average pooling layer to convert variable-length sequences into fixed-length vectors, and a recurrent layer (LSTM or GRU) to process sequence information and capture long-distance dependencies; this can be set to bidirectional to simultaneously capture contextual information. Combining CNN and RNN, local features are first extracted using CNN, then input into the RNN to process sequence relationships. Finally, a fully connected layer and an output layer are defined, flattening the output of the feature extractor before inputting it into the fully connected layer (Dense). The layer, before the final output layer, applies Dropout and BatchNormalization; Dropout: added after the fully connected layer, randomly discards a portion of neurons to prevent overfitting (e.g., Dropout(0.5)); BatchNormalization: added before or after the activation function to accelerate training and improve stability; Output layer: uses a Dense layer, with the number of neurons equal to the number of categories, and the activation function is softmax (multi-class classification) or sigmoid (multi-label classification).

[0042] Step S22: Based on the labeled software requirement data, label the requirement categories and intent tags, train the requirement classification and view recognition model, and extract the requirement text features; Step S22 specifically includes the following steps: Step S221: Perform word segmentation, stop word removal, and lemmatization on the annotated raw_text field of the requirement text, and then call the TfidfVectorizer tool to convert the annotated requirement text into a TF-IDF feature matrix X_features; Step S222: Construct a vocabulary, map the labeled requirement text to integer indices, and convert the integer sequence into a dense word vector sequence in the Embedding layer to obtain the integer label y_category for the requirement category and the integer label y_intent for the intent label; this is a key step in extracting the features of the requirement text, and the model will automatically learn the meaning of these features; finally, call train_test_split to divide the data into: training set: used for model training; validation set: used to monitor model performance during training, perform early stopping and hyperparameter tuning; test set: used for final evaluation of model performance.

[0043] Step S23: Calculate the difference between the predicted value and the true label by calling the cross-entropy loss function, and update the model parameters by calling the backpropagation algorithm.

[0044] Input a batch of training data X_batch into the model, calculate the final output predicted values ​​y_pred_category and y_pred_intent; call the cross-entropy loss function: loss_category = categorical_crossentropy(y_true_category, y_pred_category); loss_intent = categorical_crossentropy(y_true_intent, y_pred_intent); Calculate the total loss: total_loss = loss_category + loss_intent. This loss value quantifies the difference between the predicted value and the true label. Call the backpropagation algorithm: calculate the gradient of the total loss with respect to each trainable parameter of the model.

[0045] Using an optimizer (such as Adam), the model parameters are updated based on the calculated gradients, the gradients are cleared, the next batch of data is read, and this process is repeated until the entire training set (one epoch) has been traversed. The model performance is evaluated on the validation set, and a strategy (such as early stopping) is used to decide whether to start the next round of training. The final performance of the model is evaluated on the test set, and metrics such as accuracy, precision, recall, and F1-score are calculated. After training, a portion of the model (usually before the fully connected layer) can be extracted and used as a fixed demand text feature extractor. When a new demand text is input, the extractor can output a dense vector representing the semantics of the text.

[0046] Step S3: Construct a software requirement knowledge graph based on the characteristics of the requirement text, and call the visualization library to visualize the requirement classification and view recognition model in a graphical form.

[0047] Step S3 specifically includes the following steps: Step S31: Construct a software requirement knowledge graph by using the features of the requirement text as nodes and the relationships as edges; First, determine the node types: Requirement: core requirement node; Function: functional entity node; Actor: participant node (e.g., user, system); NonFunctionalAttribute: non-functional attribute node (e.g., performance, security). Second, determine the relationship types: depend_on: requirement dependency relationship; conflict_with: requirement conflict relationship; refines: requirement refinement relationship; has_function: requirement includes function; performed_by: function is performed by the participant; has_constraint: requirement has non-functional constraints. Then, node and relationship extraction is performed. Based on the structured requirement data processed in the previous steps (such as JSON data containing categories, intents, and entities), for requirement nodes: a Requirement node is created for each unique requirement text, with attributes including req_id, text, category, intent, etc.; for other nodes: functions, participants, etc., extracted from the requirement text through named entity recognition are used to create corresponding types of nodes. Relationships are extracted from the text using predefined rules; for example, if two requirement texts contain the same key functional entity, a depends_on edge is created; model-based: a relationship extraction model is trained to identify relationships between requirement statements; based on external logic: for example, edges are created based on requirement ID numbers or dependencies defined in the project documentation; if Neo4j is used to write Cypher statements, the generated nodes and edges are batch-inserted into the database.

[0048] Step S32: Invoke the graph neural network to mine the implicit relationships in the constructed software requirements knowledge graph; This embodiment utilizes graph neural networks to learn low-dimensional vector representations of nodes and edges in a graph, thereby discovering potential, undeclared demand relationships (such as hidden dependencies, conflicts, or functional clusters) in the graph. Existing edges are used as positive samples, and some non-existent edges are randomly sampled as negative samples. The model learns the embedding representation of the nodes and optimizes the decoder so that it scores positive samples higher than negative samples. After training, the model.encode method is used to obtain the final vector representation of all nodes. The score of all unconnected node pairs is calculated (using model.decode). The scores are sorted, and the node pair with the highest ranking is the implicit relationship predicted by the model. A list of potential edges is output, such as (Requirement_A, conflicts_with, Requirement_C), with confidence scores.

[0049] Step S33: Call the visualization library to read the requirement knowledge graph data from the graph database and visualize the requirement classification and view recognition model in a graphical form.

[0050] Step S33 specifically includes the following steps: Step S331: Based on the constructed Cypher query, obtain the subgraph that needs to be visualized; for example, query all requirements of a specific category and their relationships; Step S332: The high-confidence implicit relationships mined out are also used as edge data and merged with the original graph data; Step S333: Build a Flask or Django backend API to provide graph data (JSON format). The frontend requests the data via AJAX and renders it using a visualization library; interactive charts are generated directly in Jupyter Notebook or Python scripts; a requirement category view: differentiating different categories and intents by node color and shape; a relationship strength view: displaying the confidence of implicit relationships predicted by GNN by edge thickness and transparency; interactive functions: implementing features such as clicking on nodes / edges to display detailed information, dragging and dropping layouts, searching and filtering specific nodes, etc.

[0051] This embodiment uses Python and mature NLP toolkits such as NLTK and SpaCy for basic preprocessing. A custom lexicon is built, and a combination of rule-based and statistical word segmentation algorithms is used to segment the required text. Part-of-speech tagging tools are used to label the segmented results with part-of-speech tags. Regular expression matching and spell checking tools are used to identify and correct spelling errors. For unstructured text, specific text parsing rules are designed to convert it into structured formats such as JSON for storage, providing standardized input for subsequent deep learning models.

[0052] This embodiment builds a model based on the TensorFlow or PyTorch deep learning framework. It uses a collected and labeled dataset of requirements as a foundation, dividing it into training, validation, and test sets. Model structures such as CNN and RNN are constructed, and features of the required text are extracted through convolutional and recurrent layers. The difference between the predicted values ​​and the true labels is calculated using the cross-entropy loss function, and the model parameters are updated using the backpropagation algorithm. Techniques such as Dropout and Batch Normalization are employed to prevent overfitting. Hyperparameters are tuned using methods such as grid search and random search, and model performance is evaluated on the test set to ensure that the model meets the accuracy requirements for requirement classification and intent recognition.

[0053] This embodiment utilizes the Neo4j graph database to construct a requirement knowledge graph, storing requirement data and relationships within it. A relationship mining model is developed using open-source frameworks based on GNNs, such as DGL and PyG. Inputting the requirement knowledge graph data, the model is trained to learn node and edge features and uncover potential relationships. A front-end interface is developed using visualization libraries such as Echarts and D3.js, which reads the requirement knowledge graph data from the graph database and displays the requirement model graphically. Interactive operations on nodes and edges are supported, such as clicking to view detailed information and dragging to adjust the layout, facilitating intuitive user understanding of requirement relationships.

[0054] This embodiment achieves an accuracy rate of over 95% in preprocessing operations such as text cleaning, word segmentation, and part-of-speech tagging, ensuring high-quality input data. In multi-category requirement classification tasks, the model accuracy is no less than 90%, accurately classifying requirements into their corresponding categories. For requirement intent recognition, the model recall rate reaches over 85%, avoiding missing key user intents. Graph neural networks are used to mine requirement relationships, maintaining an accuracy rate of over 80%, accurately identifying dependencies, conflicts, and other relationships between requirements. Under ordinary server configurations, model training time is controlled within 24 hours, improving development efficiency. User operations on the visual interface, such as querying requirements and adjusting layouts, have a response time of no more than 3 seconds, ensuring a good interactive experience.

[0055] The other parts of this embodiment are the same as those in Embodiment 1 above, so they will not be described again.

[0056] Example 3: Based on any one of Embodiments 1-2 above, this embodiment proposes an analysis system based on natural language processing and deep learning to execute the analysis method based on natural language processing and deep learning described above; it includes a preprocessing unit, a demand classification and intent recognition unit, and a visualization unit; The preprocessing unit is used to preprocess the acquired software requirement text data to obtain a software requirement dataset and tag it with part-of-speech tags. The requirement classification and intent recognition unit is used to construct a requirement classification and view recognition model based on the labeled software requirement dataset and extract requirement text features. The visualization unit is used to construct a software requirement knowledge graph based on the characteristics of the requirement text, call the visualization library, and visualize the requirement classification and view recognition model in a graphical form.

[0057] This embodiment also proposes an electronic device, including a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, it implements the above-described analysis method based on natural language processing and deep learning.

[0058] This embodiment also proposes a computer-readable storage medium storing computer instructions; when the computer instructions are executed on the aforementioned electronic device, the aforementioned analysis method based on natural language processing and deep learning is implemented.

[0059] The other parts of this embodiment are the same as any one of the above embodiments 1-2, so they will not be described again.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. An analysis method based on natural language processing and deep learning, characterized in that, Specifically, the following steps are included: Step S1: Preprocess the acquired software requirement text data to obtain a software requirement dataset and tag it with parts of speech; Step S2: Based on the labeled software requirement dataset, construct a requirement classification and view recognition model to extract requirement text features; Step S3: Construct a software requirement knowledge graph based on the characteristics of the requirement text, and call the visualization library to visualize the requirement classification and view recognition model in a graphical form.

2. The analysis method based on natural language processing and deep learning according to claim 1, characterized in that, Step S1 specifically includes the following steps: Step S11: Call the NLP toolkit to preprocess the obtained software requirement text data; Step S12: After word segmentation and preprocessing, the software requirement text data is labeled with part-of-speech tags to obtain the labeled software requirement dataset; Step S13: Convert unstructured requirement data into structured format and store it according to the set text parsing rules.

3. The analysis method based on natural language processing and deep learning according to claim 2, characterized in that, Step S12 specifically includes the following steps: Step S121: Based on the spaCy natural language processing library, modify the Jieba word segmenter rules and construct the software requirement lexicon; Step S122: Traverse the preprocessed software requirement text data and call the pipeline segmentation of the NLP toolkit to traverse the software requirement text data; Step S123: Call token.pos_ to obtain part-of-speech tags from the segmented software requirement text data and annotate them; Step S124: Assemble the labeled software requirement data into a structured dictionary, and combine the dictionary into a list to obtain the software requirement dataset.

4. The analysis method based on natural language processing and deep learning according to claim 1, characterized in that, Step S2 specifically includes the following steps: Step S21: Based on the labeled software requirement dataset, call the neural network to build a requirement classification and view recognition model; Step S22: Based on the labeled software requirement data, label the requirement categories and intent tags, train the requirement classification and view recognition model, and extract the requirement text features; Step S23: Calculate the difference between the predicted value and the true label by calling the cross-entropy loss function, and update the model parameters by calling the backpropagation algorithm.

5. The analysis method based on natural language processing and deep learning according to claim 4, characterized in that, Step S22 specifically includes the following steps: Step S221: Use the TfidfVectorizer tool to convert the labeled text into a feature matrix; Step S222: Construct a vocabulary, map the annotated requirement text to integer indices, and convert the integer sequence into a dense word vector sequence in the Embedding layer to obtain integer labels for requirement categories and integer labels for intent labels.

6. The analysis method based on natural language processing and deep learning according to claim 1, characterized in that, Step S3 specifically includes the following steps: Step S31: Construct a software requirement knowledge graph by using the features of the requirement text as nodes and the relationships as edges; Step S32: Invoke the graph neural network to mine the implicit relationships in the constructed software requirement knowledge graph; Step S33: Call the visualization library to read the requirement knowledge graph data from the graph database and visualize the requirement classification and view recognition model in a graphical form.

7. The analysis method based on natural language processing and deep learning according to claim 6, characterized in that, Step S33 specifically includes the following steps: Step S331: Obtain the subgraph to be visualized based on the constructed Cypher query; Step S332: Merge the implicit relationships of the acquired software requirement knowledge graph as edge data with the software requirement knowledge graph to obtain the integrated model; Step S333: Build the Flask backend API. Based on the AJAX request data obtained from the frontend, call the visualization library to render the integrated model and generate interactive charts.

8. An analysis system based on natural language processing and deep learning, used to execute the analysis method based on natural language processing and deep learning as described in claim 1; characterized in that, It includes a preprocessing unit, a requirement classification and intent recognition unit, and a visualization unit; The preprocessing unit is used to preprocess the acquired software requirement text data to obtain a software requirement dataset and tag it with part-of-speech tags. The requirement classification and intent recognition unit is used to construct a requirement classification and view recognition model based on the labeled software requirement dataset and extract requirement text features. The visualization unit is used to construct a software requirement knowledge graph based on the characteristics of the requirement text, call the visualization library, and visualize the requirement classification and view recognition model in a graphical form.

9. An electronic device, characterized in that, It includes a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, it implements the analysis method based on natural language processing and deep learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions; when the computer instructions are executed on the electronic device as described in claim 9, they implement the analysis method based on natural language processing and deep learning as described in any one of claims 1-7.

Images