A big data-based software development management method

Abstract

This invention relates to the field of big data-driven software development management technology, and discloses a big data-based software development management method, including the following steps: S1, big data collection and preprocessing based on a distributed architecture; S2, requirements analysis and decomposition based on the GBDT machine learning model; S3, project planning and resource allocation based on intelligent scheduling algorithms; S4, development process control and quality inspection based on Flink real-time stream computing; S5, risk warning and handling based on the DNN deep learning model; S6, project acceptance and data review based on data mining algorithms. By adopting a distributed architecture-driven full-process data collection, standardized preprocessing, and dedicated storage management solution, this invention solves the problems of data disorder, poor reusability, and insufficient support, which lead to a lack of scientific data basis in subsequent requirements analysis, project planning, and other stages.

Description

Technical Field

[0001] This invention relates to the field of big data-driven software development management technology, specifically a big data-based software development management method. Background Technology

[0002] Software development management is a core technical field that spans the entire software development process, from requirements gathering and project planning to development implementation, quality inspection, and project acceptance. Its core objective is to use scientific management methods to coordinate key elements such as personnel, resources, and schedules, ensuring the efficient and orderly progress of software development projects and ultimately delivering software products that meet requirements and quality standards. As the scale of software development continues to expand and the complexity of requirements continues to increase, data has become a core supporting element of software development management, encompassing multiple dimensions such as requirements data, personnel data, resource data, development process data, and quality inspection data. Relying on big data technology to achieve effective collection, processing, and utilization of data throughout the entire process is an important development direction for improving the scientific and precise nature of software development management.

[0003] Existing software development management solutions suffer from significant technical shortcomings in data acquisition and processing. They lack standardized distributed acquisition architectures and efficient data transmission methods, employing fragmented acquisition approaches that only collect limited data for key stages. Furthermore, they lack standardized preprocessing using professional algorithms, and operations such as anomaly identification, missing value completion, and normalization lack unified standards. Sensitive information anonymization and data quality verification mechanisms are inadequate, and dedicated distributed databases for standardized storage and management are not established. This results in inconsistent data formats, poor standardization, and low reusability, creating industry pain points of data chaos and insufficient support. Consequently, subsequent core management stages such as requirements analysis and project planning lack reliable and unified big data support, relying solely on manual experience for judgment. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a software development and management method based on big data. This method solves the problems of fragmented data collection, lack of unified standards for preprocessing, and imperfect storage management mechanisms in existing technologies, which lead to data chaos, poor reusability, and insufficient support, resulting in a lack of scientific data basis for subsequent requirements analysis, project planning, and other stages.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a software development management method based on big data, comprising the following steps: S1. Big data collection and preprocessing based on distributed architecture: Multi-source real-time collection terminals combined with Kafka message queues are used to collect multi-dimensional raw data from the entire software development process. After preprocessing with 3σ anomaly detection, KNN missing value completion, and min-max normalization algorithm, a standardized dataset is obtained. After de-identification, the dataset is stored in HBase and HDFS distributed databases to establish a dedicated distributed big data database for software development. S2. Demand analysis and decomposition based on GBDT machine learning model: Call the standardized data in S1, complete the quantitative analysis of demand feasibility through GBDT model, sort by analytic hierarchy process, decompose into sub-demands by demand decomposition graph algorithm and generate output file. If there are infeasible or conflicting items, return to S1 to supplement data for reprocessing. S3. Project planning and resource allocation based on intelligent scheduling algorithm: Based on the output of S2, a project planning and resource allocation scheme is formulated through the fusion algorithm of genetic and particle swarm algorithms. The ARIMA algorithm is combined to predict the development time. If there are configuration problems, the system returns to S2 for adjustment and replanning. S4. Development process control and quality inspection based on Flink real-time stream computing: Develop according to the S3 plan, collect and analyze process data through the Flink framework, detect code quality by combining the AST algorithm, and carry out quality inspection by the defect density quantization algorithm. If there are any abnormalities or the inspection fails, return to the corresponding previous steps for rectification. S5. Risk warning and handling based on DNN deep learning model: Throughout the process from S2 to S4, the DNN model is used to monitor and predict risks, and the Apriori algorithm is used to match historical handling solutions. If the risk is not eliminated, the corresponding step is returned for adjustment. S6. Project acceptance and data review based on data mining algorithms: Acceptance is conducted according to standards. If it fails, it is returned to S4 for rectification. If it passes, data patterns are mined based on the FP-Growth algorithm, and the database and parameters of each intelligent model in S1 are updated in combination with the Bayesian optimization algorithm.

[0006] Preferably, in S1, the multi-source real-time acquisition terminal includes an IoT acquisition terminal, a development tool plug-in terminal, and a system docking terminal, and the acquired data covers demand data, personnel data, resource data, development process data, quality inspection data, and environmental data; During the data collection process, a unique identifier algorithm is used to uniformly record the collection time, subject, related project information, and unique identifier ID of the data source for the raw data. The data is then transmitted with low latency via a Kafka message queue.

[0007] Preferably, in step S1, the data preprocessing desensitization process adopts the national cryptographic SM4 symmetric encryption algorithm. The preprocessing process records operation logs through the ELK distributed log framework. The preprocessing effect is quantitatively verified based on the Spark distributed computing framework. The preset data integrity qualification threshold is 95%, and the coefficient of variation qualification threshold is 0.2. If the threshold is not met, the process is returned to the corresponding stage for reprocessing through the Flink stream computing task. Each data partition of the distributed database is marked with a unique collection source identifier through a hash algorithm.

[0008] Preferably, in S2, the GBDT machine learning model is trained based on the TensorFlow framework and its accuracy is optimized using cross-validation. The model is input with technical, cost, and cycle indicators as feature values ​​to obtain a quantitative score. The logic conflict of sub-requirements is identified by a graph structure conflict detection algorithm. The feasibility of the requirements is graded according to the preset scoring criteria. If the requirements are not qualified or there is a conflict, the system returns to S1 to supplement data and re-analyze.

[0009] Preferably, in S3, when formulating the project planning scheme, the historical project, personnel and resource data in S1 are combined to clarify the development cycle plan, milestone nodes, deliverable requirements and responsible entities; Personnel allocation uses the personnel load balance as the fitness function, and resource scheduling is completed based on a greedy algorithm combined with dynamic programming. The usage time, quota, and maintenance responsibility of each resource are clearly defined, and the rationality of the plan is verified through quantitative indicators.

[0010] Preferably, in S4, the development process control will perform real-time correlation and comparison analysis between the collected data and the historical data of S1 and the planning data of S3. The AST algorithm will identify code vulnerabilities, redundancies and non-standard content through syntax tree traversal. After the quality inspection is completed, the results, defect information and handling status will be recorded to form a quality inspection report. The features of high-quality code will be extracted through data mining algorithms and incorporated into the S1 experience database.

[0011] Preferably, in step S4, when the progress deviation exceeds a preset threshold, the process returns to step S3 to adjust the planning and resource allocation scheme through an intelligent scheduling algorithm; When a quality inspection fails, the root cause analysis algorithm based on fishbone diagrams and association rules is used to identify the cause and implement corrective measures. If a major technical problem occurs and the process cannot proceed, return to S2 to adjust the demand breakdown plan or return to S1 to supplement data and restart the process.

[0012] Preferably, in step S5, the DNN deep learning risk warning model is built on the PyTorch framework and optimized using stochastic gradient descent (SGD). It takes data on demand changes, schedules, quality defects, resources, and personnel as input features, clarifies the warning indicators and thresholds for five types of risks, identifies potential risks and classifies them into levels through model feature extraction and nonlinear mapping, and executes the processing plan after secondary verification by an expert system. If the standards are not met, the model parameters are optimized through transfer learning algorithms.

[0013] Preferably, in step S6, the data review extracts the experience, problems, risk handling, and acceptance data of the entire process, and after feature extraction and pattern mining, it is supplemented into the distributed database in step S1. The parameters and rules of each intelligent model are adjusted through grid search and Bayesian optimization algorithms. The review results, model update records, and adjustment plans are stored in the database through data serialization algorithms.

[0014] Preferably, in step S1, a dedicated big data database is established to store the standardized dataset after preprocessing in this step, and the standardized dataset is partitioned and stored according to data type, with each data partition corresponding to a unique source identifier.

[0015] This invention provides a software development management method based on big data. It has the following beneficial effects: 1. This invention adopts a distributed architecture-driven end-to-end data acquisition, standardized preprocessing, and dedicated storage management technical solution, which achieves the technical effect of making software development data traceable and collaboratively analyzeable across all dimensions, providing reliable and unified data support for various management stages. Compared with existing technical solutions that suffer from fragmented data acquisition, lack of unified preprocessing standards, and imperfect storage management mechanisms, this invention solves the problems of data chaos, poor reusability, and insufficient support, which lead to a lack of scientific data basis for subsequent requirements analysis, project planning, and other stages.

[0016] 2. This invention adopts a full-process intelligent management and control technology solution that deeply integrates machine learning models and intelligent algorithms. It achieves quantitative evaluation, intelligent control and automated execution of each management link, and improves the accuracy, efficiency and standardization of software development management. Compared with the existing technology that relies on human experience for management, has disconnect between links and lacks quantitative control methods, this invention solves the problems of low management efficiency, insufficient control accuracy, poor connection between links, and easy delays and substandard quality.

[0017] 3. This invention adopts a deep learning risk warning model combined with a full-process data review and optimization technical solution, which achieves accurate prediction and targeted handling of software development risks. At the same time, it promotes continuous iteration and optimization of management models and strategies, and improves the technical effect of software development risk prevention and control capabilities and overall management level. Compared with the existing technical solutions that have lagging risk warning, lack of targeted handling solutions, and no effective review and optimization mechanism, this invention solves the problems of failing to identify risks in a timely manner, poor risk handling effect, recurrence of similar problems, and difficulty in continuously improving management level. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the steps of a software development management method based on big data according to the present invention. Detailed Implementation

[0019] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see the appendix Figure 1 This invention provides a software development management method based on big data, comprising the following steps: S1. Big data collection and preprocessing based on distributed architecture: Multi-source real-time collection terminals combined with Kafka message queues are used to collect multi-dimensional raw data from the entire software development process. After preprocessing with 3σ anomaly detection, KNN missing value completion, and min-max normalization algorithm, a standardized dataset is obtained. After de-identification, the dataset is stored in HBase and HDFS distributed databases to establish a dedicated distributed big data database for software development. Among them, S1 includes an IoT acquisition terminal, a development tool plug-in terminal, and a system docking terminal. The collected data covers demand data, personnel data, resource data, development process data, quality inspection data, and environmental data. During the data collection process, a unique identifier algorithm is used to uniformly record the collection time, subject, related project information, and unique identifier ID of the data source for the raw data. The data is then transmitted with low latency via a Kafka message queue. In S1, the data preprocessing desensitization process adopts the national cryptographic SM4 symmetric encryption algorithm. The preprocessing process records operation logs through the ELK distributed log framework. The preprocessing effect is quantitatively verified based on the Spark distributed computing framework. The preset data integrity qualification threshold is 95%, and the coefficient of variation qualification threshold is 0.2. If the threshold is not met, the process is returned to the corresponding stage for reprocessing through Flink stream computing tasks. Each data partition of the distributed database is marked with a unique collection source identifier through a hash algorithm. In S1, a dedicated big data database is established to store the standardized datasets after preprocessing in this step. The standardized datasets are partitioned according to data type, and each data partition is marked with a unique source identifier. Specifically, the entire preprocessing process is logged to facilitate subsequent data traceability and post-processing optimization. A preprocessing effectiveness verification formula is also introduced, using the Spark distributed computing framework to quantitatively evaluate the quality of the preprocessed dataset, ensuring that the standardized dataset meets the needs of subsequent analysis. The core formula and its applications include: Data integrity calculation formula: ; Indicates the percentage of data integrity. This indicates the number of valid data entries after preprocessing. This formula represents the total number of original data entries. It is used to quantify the proportion of valid data in the preprocessed dataset, and intuitively reflects the completeness of the dataset after anomaly removal and missing value completion.

[0021] Meanwhile, the preset data integrity threshold is 95%. If the calculated Comp ≥ 95%, the preprocessing effect is deemed qualified and can proceed to the classification and storage stage. If Comp < 95%, the data is returned to the corresponding preprocessing stage via Flink stream computing task for reprocessing until the threshold requirement is met, thus avoiding the impact of insufficient valid data on the accuracy of subsequent big data analysis.

[0022] Data normalization verification formula: ; CV represents the coefficient of variation. This represents the standard deviation of the normalized dataset. This represents the mean of the normalized dataset. This formula is used to evaluate the effect of normalization. The smaller the coefficient of variation, the lower the dispersion of the data, the better the normalization effect, and the stronger the comparability of data of different magnitudes.

[0023] The coefficient of variation is calculated for the normalized numerical data. The preset standard for normalization is CV≤0.2. If the standard is exceeded, the normalization parameters are readjusted to ensure that raw data of different types and magnitudes can be incorporated into the same big data analysis model for collaborative analysis after processing. The system employs a 3σ anomaly detection algorithm to identify anomalies in the raw data, a KNN missing value completion algorithm to complete missing value completion, and a min-max normalization algorithm to normalize the data. Sensitive information is anonymized using the national standard SM4 symmetric encryption algorithm. Data acquisition utilizes a multi-source real-time acquisition terminal consisting of IoT acquisition terminals, development tool plugin terminals, and system interface terminals, combined with a Kafka message queue to achieve low-latency transmission. The preprocessing process records operation logs using the ELK distributed logging framework. A dedicated big data database is established using HBase and HDFS distributed databases. Standardized datasets are partitioned and stored according to data type. Each data partition is uniquely identified by a hash algorithm, and the database supports multi-node parallel access, incremental data synchronization, and disaster recovery backup.

[0024] S2. Demand analysis and decomposition based on GBDT machine learning model: Call the standardized data in S1, complete the quantitative analysis of demand feasibility through GBDT model, sort by analytic hierarchy process, decompose into sub-demands by demand decomposition graph algorithm and generate output file. If there are infeasible or conflicting items, return to S1 to supplement data for reprocessing. In S2, the GBDT machine learning model is trained based on the TensorFlow framework and uses cross-validation to optimize accuracy. The model is input with technical, cost, and cycle indicators as feature values ​​to obtain a quantitative score. The logic conflict of sub-requirements is identified by the graph structure conflict detection algorithm. The feasibility of the requirements is graded according to the preset scoring criteria. If the requirements are unqualified or conflict exists, the model is returned to S1 to supplement data and re-analyze. Specifically, after the requirements analysis and breakdown are completed, a requirements review is conducted by multiple parties, and the output documents are optimized based on the review comments. For the quantitative verification of requirements feasibility and prioritization, core formulas and applications are supplemented to avoid repetition with previous sections, specifically including: Demand Feasibility Comprehensive Scoring Formula: ; This indicates the overall score for the feasibility of the demand. Indicates the technical feasibility score. Indicates the cost feasibility score. Indicates the feasibility score of the cycle; , , These are the weights for the three feasibility factors, which can be dynamically adjusted according to the project type.

[0025] The default score is Feas ≥ 80, which indicates the requirement is feasible; Feas ≤ 60 < 80 indicates the requirement needs optimization and adjustment; and Feas < 60 indicates the requirement is infeasible. If a requirement is found to have Feas < 60 during the review, or if a logical conflict is identified in a sub-requirement through the graph structure conflict detection algorithm, the process returns to S1 to supplement relevant data and recalculates the score until Feas ≥ 80 and there are no conflicting items, thus ensuring the scientific validity and feasibility of the requirement analysis results.

[0026] Demand Priority Calculation Formula: ; This represents the priority score of the i-th requirement. This indicates the business value score of the requirement. This indicates the intensity score of user demand. The score indicates the difficulty of development. Indicates the cost percentage score; , , and The weights for each indicator are determined by the project team in conjunction with business objectives.

[0027] according to The scores are used to rank all requirements. The top 30% are core requirements, 30%-70% are secondary requirements, and the bottom 30% are optimization requirements. The ranking results are stored synchronously in the S1 database as the core basis for subsequent project planning and resource allocation, ensuring the objectivity of priority ranking and avoiding human experience bias. The big data analysis model uses a GBDT machine learning model trained on the TensorFlow framework and optimized for accuracy through cross-validation. Technical, cost, and cycle indicators are used as feature values ​​to input into the model to complete quantitative analysis. The demand priority ranking uses the analytic hierarchy process to determine the indicator weights. The overall demand is decomposed into sub-demands using a demand decomposition graph algorithm, and a dependency graph of sub-demands is constructed.

[0028] S3. Project planning and resource allocation based on intelligent scheduling algorithm: Based on the output of S2, a project planning and resource allocation scheme is formulated through the fusion algorithm of genetic and particle swarm algorithms. The ARIMA algorithm is combined to predict the development time. If there are configuration problems, the system returns to S2 for adjustment and replanning. In S3, when formulating project planning schemes, the historical project, personnel and resource data in S1 are combined to clarify the development cycle plan, milestone nodes, deliverable requirements and responsible parties; Personnel allocation is based on the fitness function of personnel load balance, and resource scheduling is completed based on a greedy algorithm combined with dynamic programming. The usage time, quota and maintenance responsibility of each resource are clearly defined, and the rationality of the plan is verified by quantitative indicators. Specifically, after project planning and resource allocation are completed, a feasibility verification formula is introduced to quantitatively evaluate the rationality of the schedule, personnel allocation, and resource scheduling, supplementing the quantitative verification logic not mentioned above. This includes: Personnel load balance formula: ; Indicates the degree of personnel workload balance. Let represent the average daily workload of the i-th developer. Indicates the maximum load. Indicates minimum load. This represents the average workload of all developers; The closer it is to 1, the more balanced the workload of the personnel and the more reasonable the allocation of resources.

[0029] The default value of Bal ≥ 0.8 is considered a qualified load balance. If Bal < 0.8, it indicates a problem of personnel load imbalance. The intelligent scheduling algorithm will adjust the personnel allocation plan and rematch sub-requirements with development personnel until Bal ≥ 0.8, so as to avoid schedule delays or waste of human resources due to load imbalance.

[0030] Resource utilization rate formula: ; This represents the average utilization rate of core resources. This represents the actual resource usage during time period t. This formula represents the maximum available resources, and T represents the total number of project periods. It is used to quantitatively evaluate the rationality of resource scheduling and avoid resource idleness or over-occupancy.

[0031] The preset reasonable range for resource utilization is 60%-80%. If Uti < 60%, it indicates that resources are seriously idle, and the resource usage period should be adjusted or the allocation of sub-demands should be increased. If Uti > 80%, it indicates that resources are over-occupied and there is a risk of resource conflict. The intelligent scheduling algorithm will return an adjusted resource scheduling plan to supplement resources or optimize the usage period to ensure that the resource utilization is within a reasonable range.

[0032] Formula for the rationality of progress buffer: ; Indicates the progress buffer ratio. Indicates the overall planning period of the project. This represents the pure development cycle; this formula is used to evaluate the rationality of the buffer time setting and to balance schedule flexibility and cycle control.

[0033] Preset The reasonable range is 10%-15%. If Buf < 10%, the buffer time is insufficient and cannot cope with sudden schedule delays; if Buf > 15%, the buffer time is too long, resulting in wasted time. In both cases, the schedule needs to be adjusted to ensure that the buffer time is reasonable. The ARIMA time series algorithm is used to predict the time consumption of each stage of software development and formulate a development cycle plan. The allocation of personnel and resources adopts an intelligent scheduling algorithm that combines genetics and particle swarm optimization. The personnel allocation uses the personnel load balance as the fitness function, and the resource scheduling is completed based on a greedy algorithm combined with dynamic programming. The feasibility quantification and verification of the solution is carried out based on the Spark distributed computing framework. The unmet targets are re-iterated and optimized through intelligent scheduling algorithm.

[0034] S4. Development process control and quality inspection based on Flink real-time stream computing: Develop according to the S3 plan, collect and analyze process data through the Flink framework, detect code quality by combining the AST algorithm, and carry out quality inspection by the defect density quantization algorithm. If there are any abnormalities or the inspection fails, return to the corresponding previous steps for rectification. In S4, the development process control will conduct real-time correlation and comparison analysis of the collected data with the historical data of S1 and the planning data of S3. The AST algorithm will identify code vulnerabilities, redundancies and non-standard content through syntax tree traversal. After the quality inspection is completed, the results, defect information and handling status will be recorded to form a quality inspection report. The features of high-quality code will be extracted through data mining algorithms and incorporated into the S1 experience database. In S4, if the schedule deviation exceeds the preset threshold, return to S3 to adjust the planning and resource allocation scheme through the intelligent scheduling algorithm; When a quality inspection fails, the root cause analysis algorithm based on fishbone diagrams and association rules is used to identify the cause and implement corrective measures. If a major technical problem occurs and the process cannot proceed, return to S2 to adjust the demand breakdown plan or return to S1 to supplement data and restart the process. Specifically, based on progress control, code quality, and defect management, a quantitative control formula will be added, and the logic for exception detection and quality assessment not covered previously will be improved. This includes: Schedule Deviation Warning Formula: ; Indicates the schedule deviation rate. Indicates the actual progress completed. This indicates the planned progress; a positive Dev value indicates that the progress is ahead of schedule, while a negative value indicates that the progress is behind schedule.

[0035] The preset warning threshold is ±5%, meaning that a progress warning is triggered when Dev < -5% or Dev > 5%. For delay warnings, the duration of the delay impact is calculated by combining historical S1 data. ,like > Buffer time, return to S3 to adjust the plan through intelligent scheduling algorithm; for early warning, assess whether there are potential quality problems and avoid blindly rushing to complete the work.

[0036] Overall code quality scoring formula: ; Q represents the overall code quality score, A represents the code standardization score, B represents the code readability score, and C represents the code security score. , , The weights are denoted by , with security having the highest weight.

[0037] The code with a Q value of ≥ 85 is considered a high-quality code, a code with a Q value of 70 ≤ Q < 85 is considered a qualified code, and a code with a Q value of < 70 is considered a substandard code. Substandard codes must be rectified and resubmitted for testing. Qualified codes can proceed to the next stage. High-quality codes will have their features extracted using data mining algorithms and will be included in the S1 experience database to provide a reference for subsequent development.

[0038] Defect density calculation formula: ; Indicates defect density, This represents the total number of defects detected, and LOC represents the total number of lines of code. This formula is used to quantitatively assess the degree of potential software quality issues; the lower the defect density, the more guaranteed the quality.

[0039] Based on the project type, a defect density threshold is preset. If the detected D exceeds the threshold, a quality warning is triggered, the defect-concentrated module is analyzed, and the corresponding development stage is returned for comprehensive rectification. At the same time, the parameters of the quality inspection model are adjusted to improve the accuracy of subsequent inspections. Development process data is collected through a Kafka message queue and then transmitted to the Flink real-time stream computing framework for real-time correlation and comparison analysis with S1 historical data and S3 planning data to achieve real-time progress control. Code quality inspection uses the AST abstract syntax tree algorithm to identify code problems through syntax tree traversal. Multi-dimensional quality inspection uses a defect density quantization algorithm. When quality inspection fails, the root cause analysis algorithm of defect is used to identify the cause of the defect through fishbone diagram and association rule mining.

[0040] S5. Risk warning and handling based on DNN deep learning model: Throughout the process from S2 to S4, the DNN model is used to monitor and predict risks, and the Apriori algorithm is used to match historical handling solutions. If the risk is not eliminated, the corresponding step is returned for adjustment. In S5, the DNN deep learning risk warning model is built on the PyTorch framework and uses stochastic gradient descent (SGD) optimization. It takes data on demand changes, schedules, quality defects, resources, and personnel as input features, clarifies the warning indicators and thresholds for five types of risks, identifies potential risks and classifies them into levels through model feature extraction and nonlinear mapping, and executes the processing plan after secondary verification by the expert system. If the standard is not met, the model parameters are optimized through transfer learning algorithm. Specifically, this involves establishing a risk ledger and supplementing it with formulas for quantitative risk assessment and early warning accuracy verification, improving the quantitative logic of risk management, and supplementing content not covered previously, including: Risk level quantification calculation formula: ; R represents the risk quantification value, P represents the probability of risk occurrence, and I represents the degree of risk impact. Risk levels are classified according to the R value: R≥8 is Level 1 risk, 6≤R<8 is Level 2 risk, 3≤R<6 is Level 3 risk, and R<3 is Level 4 risk.

[0041] The system calculates the R-value for various risks in real time and triggers corresponding processing procedures based on the level. For example, in the risk of employee turnover, the turnover intention score of core developers corresponds to the P-value, and the irreplaceability of their positions corresponds to the I-value. If R≥8, the personnel backup plan is immediately activated, and the workload is adjusted to reduce the probability of turnover.

[0042] Risk warning accuracy verification formula: ; Indicates the accuracy of risk warning. This indicates the number of risks that are accurately predicted. This represents the total number of warnings; this formula is used to evaluate the accuracy of the risk warning model and facilitates subsequent model optimization.

[0043] The preset early warning accuracy threshold is 80%, and calculations are performed periodically. Value, if If the accuracy is less than 80%, it indicates that the model's early warning accuracy is insufficient. Analyze the reasons for false alarms and missed alarms, adjust the weights of the model's input indicators or the early warning threshold through transfer learning algorithms, and update the historical risk data in S1 to improve the model's training effect and ensure the accuracy of risk early warning.

[0044] Formula for verifying the effectiveness of risk management: ; Indicates the percentage of risk management effectiveness. This represents the quantified risk value before processing. This represents the quantified risk value after processing; The higher the value, the better the processing effect.

[0045] Preset ≥70% is considered acceptable for processing. If the success rate is less than 70%, it indicates that the treatment plan has not achieved the expected results. The treatment plan should be revised or the corresponding preliminary steps adjusted until the desired outcome is achieved. ≥70%, ensuring complete elimination of risk; The big data risk early warning model adopts a DNN deep learning model built on the PyTorch framework and optimized by stochastic gradient descent (SGD). It takes data on demand changes, schedules, quality defects, resources, and personnel as input features, and analyzes data change trends and abnormal features through model feature extraction and nonlinear mapping. The risk handling plan is generated by calling historical risk handling data through the Apriori association rule mining algorithm, and the plan must be verified twice by the expert system before execution. When the model's early warning accuracy is not up to standard, the model parameters are optimized through transfer learning algorithms.

[0046] S6. Project acceptance and data review based on data mining algorithms: Acceptance is conducted according to standards. If it fails, it is returned to S4 for rectification. If it passes, data patterns are mined based on the FP-Growth algorithm, and the database and parameters of each intelligent model in S1 are updated in combination with the Bayesian optimization algorithm. In S6, the data review extracts the experience, problems, risk handling, and acceptance data of the entire process. After feature extraction and pattern mining, it is added to the S1 distributed database. The parameters and rules of each intelligent model are adjusted through grid search and Bayesian optimization algorithms. The review results, model update records, and adjustment plans are stored in the database through data serialization algorithms. Specifically, after project acceptance, a formula for quantifying the effectiveness of the post-mortem analysis will be added to assess its supporting value for subsequent projects and improve the closed-loop logic of the post-mortem analysis. This includes: Formula for the value of data reuse in post-mortem analysis: ; Val represents the value of reusing retrospective data. This indicates the number of reusable experiences and solutions. This represents the total amount of data reviewed, and $$\bar{E}$$ represents the average performance score of the reusable data. A Val score of ≥7 is considered a satisfactory post-mortem analysis, indicating that the post-mortem data has high reusability. If Val <7, the post-mortem data needs to be re-examined, and reusable experiences and solutions should be added to ensure that the post-mortem data can provide effective support for similar projects in the future. The Val value is synchronously stored in the S1 post-mortem partition as a reference for subsequent post-mortem optimization.

[0047] Formula for verifying model optimization effect: ; Imp represents the model optimization improvement rate. This represents the performance metrics before model optimization. This represents the performance metric after model optimization; Imp is positive, indicating improved model performance, and the larger the value, the better the optimization effect.

[0048] For the adjusted big data analysis model, risk warning model, and quality inspection model, calculate the Imp value respectively. The preset Imp ≥ 10% is considered to be the optimization qualified. If Imp < 10% for a certain model, it means that the optimization effect is not good. Re-analyze and review the data, adjust the model parameters and rules until Imp ≥ 10%, and ensure that the model performance is continuously improved to achieve closed-loop optimization of software development management.

[0049] Project goal achievement rate formula: ; Ach represents the project goal achievement rate, and n represents the number of project goals. This represents the weight of the i-th objective. This indicates the achievement status of the i-th goal; this formula is used to quantitatively evaluate the overall implementation effect of the project and provide a core basis for review and summary.

[0050] When reviewing the project, calculate the Ach value. If Ach ≥ 90%, it indicates that the overall implementation of the project was excellent, and the focus should be on extracting the advantages and experiences. If 70% ≤ Ach < 90%, it indicates that some objectives were not achieved, and the focus should be on analyzing the reasons for the non-achievement and formulating improvement measures. If Ach < 70%, a comprehensive review of the entire process is required to identify core issues, adjust subsequent project management strategies, and avoid the recurrence of similar problems. Project acceptance uses a quantitative scoring algorithm based on acceptance indicators to conduct full-dimensional scoring; full-process data extraction and pattern mining uses the FP-Growth association rule mining algorithm; parameter and rule adjustment of each intelligent model uses grid search and Bayesian optimization algorithms; review results, model update records and adjustment plans are stored in S1's HBase and HDFS distributed databases through data serialization algorithms.

[0051] Establish a dedicated big data database to store the standardized datasets preprocessed in this step, and store the standardized datasets in partitions according to data types, with each data partition labeled with a unique source identifier.

[0052] The following is a description with reference to specific embodiments: Example 1: A software development management method based on big data, for small to medium-sized conventional software development scenarios, including the following steps: S1. Big Data Acquisition and Preprocessing Based on Distributed Architecture: A simple multi-source data acquisition terminal is built, employing a combination of IoT acquisition terminals, development tool plugin terminals, and system interface terminals. This combines terminal acquisition with manual data entry to comprehensively collect multi-dimensional raw data from the entire software development process. Development tool plugins collect requirement data, personnel data, resource data, development process data, quality inspection data, and environmental data. During the acquisition process, the acquisition time, acquisition subject, and related project information are uniformly recorded for all raw data. Low-latency data transmission is achieved through a Kafka message queue. The collected raw data underwent preprocessing: A 3σ anomaly detection algorithm was used to remove abnormal data exceeding the normal range in server load and noisy data with incorrect formatting in bug records; the KNN missing value completion algorithm was used to complete missing technical fields in the developer skill list using the average skill values ​​of employees in the same position; the min-max normalization algorithm was used to convert time records of different formats into a unified format, and data of different magnitudes such as server load and network latency were normalized to the [0,1] interval to obtain a standardized dataset; sensitive information such as developer contact information and core customer requirement parameters was anonymized using the national cryptographic SM4 symmetric encryption algorithm, and then stored in HBase and HDFS distributed databases according to data type. The databases were partitioned according to data type, and each partition was marked with a unique source identifier using a hash algorithm. The preprocessing process recorded operation logs using the ELK distributed logging framework, and the preprocessing effect was verified based on the Spark distributed computing framework. The data integrity compliance rate was 98%, and the coefficient of variation was 0.15.

[0053] S2. Requirements Analysis and Decomposition Based on GBDT Machine Learning Model: The preprocessed, standardized requirements data from S1 is used as input. A GBDT machine learning model, trained using the TensorFlow framework and optimized for accuracy through cross-validation, is used as the input feature values ​​for technical, cost, and cycle feasibility analysis of the overall requirements of the business management system. The analytic hierarchy process (AHP) is used to determine the weights of the indicators, prioritizing the overall requirements into three levels: core requirements, secondary requirements, and optimization requirements. The requirements are then decomposed into four independent but interconnected sub-requirements—user management, data entry, report generation, and access control—using a requirement decomposition graph algorithm. A dependency graph is constructed to clarify the development scope, acceptance criteria, and dependencies of each sub-requirement, generating a requirements analysis report and a list of sub-requirements. No logical conflicts were identified in the sub-requirements using a graph structure conflict detection algorithm. The overall requirements feasibility score is 86. No infeasible or conflicting items were found in this requirements analysis, and the process proceeds directly to step S3.

[0054] S3. Project Planning and Resource Allocation Based on Intelligent Scheduling Algorithms: Based on the sub-requirement list in S2, the personnel data, resource data, and historical project data in S1 are used to formulate a 3-month development cycle plan based on the ARIMA time series algorithm, clarifying milestone nodes, deliverable requirements, and responsible parties. According to the development difficulty and technical requirements of each sub-requirement, combined with the skills and workload data of the developers, a genetic and particle swarm optimization fusion intelligent scheduling algorithm is used to assign corresponding developers to each sub-requirement, using personnel load balance as the fitness function to ensure personnel load balance. Based on a greedy algorithm combined with dynamic programming, 2 servers, 3 sets of development tools, and 1 set of testing tools are scheduled, clarifying the usage time, usage quota, and maintenance responsibilities of the development servers, forming a resource allocation plan. Quantitative verification is performed using the Spark distributed computing framework, with a personnel load balance of 0.85, resource utilization of 72%, and a progress buffer ratio of 12%. No resource shortages or unreasonable personnel configurations were found in this planning. After generating the project plan, proceed to step S4.

[0055] S4. Development Process Control and Quality Inspection Based on Flink Real-Time Stream Computing: Development is initiated according to the plan in S3. Code submission data, development progress data, and problem feedback data during the development process are collected through the Kafka message queue and transmitted to the Flink real-time stream computing framework. Real-time correlation and comparison analysis are performed with historical data in S1 and planned data in S3 to complete process control and quality inspection: real-time control of development progress, with no deviations; based on the AST abstract syntax tree algorithm, the code in the development process is checked for standardization, readability, and security. Three code redundancies and one security vulnerability are identified through syntax tree traversal and fed back to the corresponding developers for modification; after the development of each sub-requirement is completed, multi-dimensional quality inspection is carried out through the defect density quantification algorithm, and the inspection results, defect information, and handling status are recorded to form a quality inspection report; if the report generation module fails the test, the defect root cause analysis algorithm based on fishbone diagram and association rule mining is used to identify the defect cause and return to the corresponding development stage for rectification. After rectification, the re-test is passed and proceeds to step S5.

[0056] S5. Risk Warning and Handling Based on DNN Deep Learning Model: Throughout the entire process from S2 to S4, various types of data from S1 are called up in real time and transmitted to a DNN deep learning risk warning model built on the PyTorch framework and optimized using stochastic gradient descent (SGD). Through model feature extraction and nonlinear mapping analysis, data change trends and abnormal characteristics are analyzed to monitor and predict various risks. When building the model, the warning indicators and thresholds for requirement change risks, schedule delay risks, and quality defect risks are clearly defined. The schedule data and defect data in S1 are analyzed in real time. If no data anomalies or potential risks are found, no risk handling is required, and the process proceeds directly to step S6.

[0057] S6. Project Acceptance and Data Review Based on Data Mining Algorithms: Following the acceptance criteria in S2 and the project plan in S3, project acceptance was conducted. Acceptance data was collected using a quantitative scoring algorithm based on acceptance indicators and scored across all dimensions. This data was then compared with historical acceptance data in the standardized dataset in S1 to determine if each sub-requirement and the overall software met the acceptance criteria, thus passing the acceptance test. After acceptance, a review of the entire software development process was performed. Based on the FP-Growth association rule mining algorithm, experience data, problem data, risk handling data, and acceptance data were extracted from the entire process, and feature extraction and pattern mining were completed, supplementing the data to the HBase and HDFS distributed databases in S1. The review data was analyzed, and the parameters and rules of the GBDT requirements analysis model and the DNN risk warning model were adjusted using grid search and Bayesian optimization algorithms. The review results, model update records, and adjustment plans were stored in a big data database using a data serialization algorithm. The model optimization improvement rate was 13%, and the review data reuse value was 7.8 points, completing the review.

[0058] Example 2: A software development management method based on big data, for large-scale, complex, iterative software development scenarios, including the following steps: S1. Big Data Acquisition and Preprocessing Based on Distributed Architecture: A comprehensive, multi-source data acquisition terminal system is built, employing a combination of IoT acquisition terminals, development tool plug-in terminals, and system interface terminals. This system utilizes automatic monitoring, terminal acquisition, and system integration to comprehensively collect multi-dimensional raw data from the entire software development process. The system connects to the development platform to collect requirement data, personnel data, resource data, development process data, quality inspection data, and environmental data. During the acquisition process, all raw data is uniformly recorded with acquisition time, acquisition subject, associated project information, and iteration version information. Low-latency data transmission is achieved through a Kafka message queue. The collected raw data underwent preprocessing: Extreme outliers in environmental data and invalid records in development process data were removed using the 3σ anomaly detection algorithm; missing attendance records in personnel attendance data were filled using the KNN missing value completion algorithm, employing the average attendance of the same position during the same period; data of different formats and magnitudes were converted to a unified specification using the min-max normalization algorithm to obtain a standardized dataset; sensitive information such as developer privacy information and core system technical parameters was anonymized using the national cryptographic SM4 symmetric encryption algorithm, and then stored in HBase and HDFS distributed databases according to data type and iteration version. The databases were partitioned according to data type, and each partition was marked with a unique source identifier using a hash algorithm. The preprocessing process recorded operation logs using the ELK distributed logging framework, and the preprocessing effect was verified using the Spark distributed computing framework, achieving a data integrity compliance rate of 99% and a coefficient of variation of 0.12. S2. Requirements Analysis and Decomposition Based on GBDT Machine Learning Model: The preprocessed, standardized requirements data from S1 is used as input. A GBDT machine learning model, trained using the TensorFlow framework and optimized for accuracy through cross-validation, is input. Technical, cost, and cycle indicators are used as feature values ​​to quantitatively analyze the overall requirements of the large-scale system software in terms of technical feasibility, cost feasibility, and cycle feasibility. The weights of the indicators are determined using the Analytic Hierarchy Process (AHP), and the overall requirements are prioritized. The overall requirements are decomposed into eight core sub-requirements, including core transactions, user profiling, data analysis, and system maintenance, using a requirements decomposition graph algorithm. Each sub-requirement is further decomposed into several sub-modules, constructing a sub-requirement dependency graph to clarify the development scope, acceptance criteria, and dependencies of each sub-requirement and sub-module, generating a requirements analysis report and a list of sub-requirements. No logical conflicts were identified using the graph structure conflict detection algorithm, but initial analysis revealed an infeasible item due to insufficient personnel resources. The overall requirements feasibility score is 58. Returning to S1 to collect additional personnel recruitment data, the data is re-entered into the GBDT model for scoring, resulting in a score of 83. Proceed to step S3. S3. Project Planning and Resource Allocation Based on Intelligent Scheduling Algorithms: Based on the sub-requirement list in S2, the personnel data, resource data, and historical project data in S1 are used to formulate a 12-month development cycle plan based on the ARIMA time series algorithm, clarifying the milestone nodes, deliverable requirements, and responsible parties for three iterations; based on the development difficulty and technical requirements of each sub-requirement and sub-module, combined with the skill specialties and workload data of the developers, a genetic and particle swarm optimization intelligent scheduling algorithm is used to assign corresponding developers to each sub-requirement, using the personnel load balance as the fitness function to ensure personnel load balance; based on a greedy algorithm combined with dynamic programming, 10 servers, 15 sets of development tools, 5 sets of testing tools, and 2 sets of operation and maintenance tools are scheduled, clarifying the usage time, usage amount, and maintenance responsibility of each resource, forming a resource allocation plan; based on the Spark distributed computing framework, quantitative verification shows a personnel load balance of 0.88, a resource utilization rate of 75%, and a progress buffer ratio of 13%. No other configuration issues were found in this planning. After generating the project planning document, proceed to step S4. S4. Development Process Control and Quality Inspection Based on Flink Real-Time Stream Computing: Development was initiated according to the plan in S3, progressing in three iterations. Code commit data, development progress data, and issue feedback data were collected during the development process through a Kafka message queue and transmitted to the Flink real-time stream computing framework. Real-time correlation and comparison analysis was performed with historical data from S1 and planned data from S3 to complete process control and quality inspection. Real-time monitoring of development progress was implemented. In version V1.0, a progress deviation of over 5 days was found in the user profiling module, with a deviation rate of -7%. Progress was restored after adjusting personnel configuration using an intelligent scheduling algorithm. The AST (Abstract Syntax Tree) algorithm was also implemented. The code is tested for standardization, readability, and security during development. Each iteration identifies several code vulnerabilities, redundancies, and non-standard content through syntax tree traversal and provides feedback to the corresponding developers for modification. After each submodule and each iteration version is completed, multi-dimensional quality testing is conducted using a defect density quantification algorithm. The test results, defect information, and handling status are recorded to generate a quality test report. If the V2.0 version data analysis module fails the quality test, the root cause analysis algorithm based on fishbone diagrams and association rules is used to identify the defect and the module is returned to the corresponding development stage for rectification. After rectification, the module passes the retest. After all three iterations of development and testing are completed, the process proceeds to step S5. S5. Risk Warning and Handling Based on DNN Deep Learning Model: Throughout S2 to S4, various data from S1 are accessed in real time and transmitted to a DNN deep learning risk warning model built on the PyTorch framework and optimized using stochastic gradient descent (SGD). The model extracts features and analyzes data trends and anomalies through nonlinear mapping to monitor and predict various risks. When building the model, warning indicators and thresholds for various risks are clearly defined. Real-time analysis of various data from S1 reveals that, during the development of version V3.0, a core developer's workload was consistently too high. The model calculated a risk quantification value R=8.5, triggering a Level 1 personnel turnover risk warning. A targeted solution is generated by using the Apriori association rule mining algorithm to access historical personnel turnover risk handling data. After secondary verification by the expert system, the solution is executed. Continuous monitoring follows the execution of the solution, and the risk quantification value R is recalculated to 2.3, with a risk handling effectiveness percentage of 73%, confirming risk elimination. No other major risks occurred throughout the process, proceeding to step S6. S6. Project Acceptance and Data Review Based on Data Mining Algorithms: Following the acceptance criteria of S2 and the project plan of S3, project acceptance was conducted iteratively, culminating in a final overall acceptance. Acceptance data for each iteration and the overall software was collected using a quantitative scoring algorithm based on acceptance indicators, and scored across all dimensions. This data was compared with historical acceptance data in the standardized dataset of S1 to determine if each iteration and the overall software met the acceptance criteria, thus passing the acceptance. After acceptance, a review of the entire software development process was conducted. Experience data, problem data, risk handling data, and acceptance data were extracted from the entire process using the FP-Growth association rule mining algorithm, and feature extraction and pattern mining were performed, supplementing the data to the HBase and HDFS distributed databases of S1. The review data was analyzed, and the parameters and rules of various models were adjusted using grid search and Bayesian optimization algorithms. The review results, model update records, and adjustment plans were stored in a big data database using a data serialization algorithm. The model optimization improvement rate was 15%, the review data reuse value was 8.2 points, and the project goal achievement rate was 92%, thus completing the review.

[0059] Example 3: A software development management method based on big data, for lightweight and rapid software development scenarios, including the following steps: S1. Big Data Acquisition and Preprocessing Based on Distributed Architecture: A lightweight multi-source data acquisition terminal is built, employing a combination of IoT acquisition terminals, development tool plugin terminals, and system interface terminals. The primary method is automatic terminal acquisition, supplemented by manual input, to collect multi-dimensional raw data from the entire software development process. This includes collecting requirement data, personnel data, resource data, development process data, quality inspection data, and environmental data through development tool plugins. During the acquisition process, the acquisition time, acquisition subject, and related project information are uniformly recorded for all raw data, and low-latency data transmission is achieved through a Kafka message queue. The collected raw data underwent preprocessing: 3σ anomaly detection algorithm was used to remove abnormal fluctuations in server load and invalid information from bug logs; KNN missing value completion algorithm was used to manually supplement missing developer work records; min-max normalization algorithm was used to convert raw data of different formats and magnitudes into a standardized dataset; and the developer privacy information was anonymized using the national cryptographic SM4 symmetric encryption algorithm before being categorized by data type and stored in HBase and HDFS distributed databases. The databases were partitioned by data type, and each partition was uniquely identified by a hash algorithm. Operation logs were recorded during the preprocessing process using the ELK distributed logging framework. The preprocessing effect was verified using the Spark distributed computing framework, achieving a data integrity compliance rate of 96% and a coefficient of variation of 0.18. S2. Requirements Analysis and Decomposition Based on GBDT Machine Learning Model: The preprocessed, standardized requirements data from S1 is used as input. A lightweight GBDT machine learning model, trained using the TensorFlow framework and optimized for accuracy through cross-validation, is employed. Technical, cost, and cycle indicators are used as feature values ​​to quantitatively analyze the overall requirements of the query tool in terms of technical, cost, and cycle feasibility. The weights of the indicators are determined using the Analytic Hierarchy Process (AHP), and the overall requirements are prioritized into core and optimized requirements. The overall requirements are decomposed into four simple sub-requirements: data import, query function, result export, and interface display. A dependency graph of these sub-requirements is constructed, clarifying the development scope, acceptance criteria, and dependencies of each sub-requirement. A requirements analysis report and a list of sub-requirements are generated. No logical conflicts were identified in the sub-requirements using the graph structure conflict detection algorithm. The overall feasibility score is 82. No infeasible or conflicting items were found in this requirements analysis, and the process proceeds directly to step S3. S3. Project Planning and Resource Allocation Based on Intelligent Scheduling Algorithms: Based on the sub-requirement list in S2, the personnel data, resource data, and historical project data in S1 are used to formulate a one-month development cycle plan based on the ARIMA time series algorithm, clarifying milestone nodes, deliverable requirements, and responsible parties. According to the development difficulty and technical requirements of each sub-requirement, combined with the skills and workload data of the developers, a lightweight intelligent scheduling algorithm combining genetics and particle swarm optimization is used to assign corresponding developers to each sub-requirement, using personnel load balance as the fitness function to ensure personnel load balance. Based on a greedy algorithm combined with dynamic programming, one server, two sets of development tools, and one set of simple testing tools are scheduled, clarifying the server usage time, usage quota, and maintenance responsibilities, forming a resource allocation plan. Quantitative verification is performed using the Spark distributed computing framework, with a personnel load balance of 0.83, resource utilization of 68%, and a progress buffer ratio of 11%. No configuration issues were found in this planning. After generating the project plan, proceed to step S4. S4. Development Process Control and Quality Inspection Based on Flink Real-Time Stream Computing: Development is initiated according to the plan in S3. Code submission data, development progress data, and problem feedback data during the development process are collected through the Kafka message queue and transmitted to the lightweight Flink real-time stream computing framework. Real-time correlation and comparison analysis are performed with historical data in S1 and planned data in S3 to complete process control and quality inspection: real-time control of development progress, with no deviations; simple standardization and security checks are performed on the code during the development process based on the lightweight AST abstract syntax tree algorithm. Two non-standard code contents are identified through syntax tree traversal and fed back to the corresponding developers for modification; after the development of each sub-requirement is completed, multi-dimensional quality inspection is carried out through the lightweight defect density quantification algorithm, and the inspection results, defect information, and handling status are recorded to form a quality inspection report; if the result export module fails the test, the cause of the defect is identified by the lightweight defect root cause analysis algorithm based on fishbone diagram and association rule mining, and returned to the corresponding development stage for rectification. After rectification, the re-test is passed and proceeds to step S5; S5. Risk Warning and Handling Based on DNN Deep Learning Model: Throughout S2 to S4, various data from S1 are accessed in real time and transmitted to a lightweight DNN deep learning risk warning model built on the PyTorch framework and optimized using stochastic gradient descent (SGD). Through model feature extraction and nonlinear mapping analysis, data change trends and abnormal characteristics are identified to monitor and predict various risks. When building the model, warning indicators and thresholds for schedule delay risks and quality defect risks are clearly defined. If no data anomalies or potential risks are found in real time, no risk handling is required, and the process proceeds directly to step S6. S6. Project Acceptance and Data Review Based on Data Mining Algorithms: Following the acceptance criteria in S2 and the project plan in S3, project acceptance was conducted. Acceptance data was collected and scored using a lightweight acceptance indicator quantitative scoring algorithm. This data was then compared with historical acceptance data in the standardized dataset in S1 to determine if each sub-requirement and the overall software met the acceptance criteria, thus passing the acceptance test. After acceptance, a review of the entire software development process was conducted. Experience data, problem data, and acceptance data were extracted from the entire process using a lightweight FP-Growth association rule mining algorithm, and feature extraction and pattern mining were performed, supplementing the data to the HBase and HDFS distributed databases in S1. The review data was analyzed, and the parameters and rules of the big data analysis model and quality inspection model were adjusted using grid search and Bayesian optimization algorithms. The review results, model update records, and adjustment plans were stored in the big data database using a data serialization algorithm. The model optimization improvement rate was 11%, and the review data reuse value was 7.3 points, completing the review.

[0060] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A software development management method based on big data, characterized in that, Includes the following steps: S1. Big data collection and preprocessing based on distributed architecture: Multi-source real-time collection terminals combined with Kafka message queues are used to collect multi-dimensional raw data from the entire software development process. After preprocessing with 3σ anomaly detection, KNN missing value completion, and min-max normalization algorithm, a standardized dataset is obtained. After de-identification, the dataset is stored in HBase and HDFS distributed databases to establish a dedicated distributed big data database for software development. S2. Demand analysis and decomposition based on GBDT machine learning model: Call the standardized data in S1, complete the quantitative analysis of demand feasibility through GBDT model, sort by analytic hierarchy process, decompose into sub-demands by demand decomposition graph algorithm and generate output file. If there are infeasible or conflicting items, return to S1 to supplement data for reprocessing. S3. Project planning and resource allocation based on intelligent scheduling algorithm: Based on the output of S2, a project planning and resource allocation scheme is formulated through the fusion algorithm of genetic and particle swarm algorithms. The ARIMA algorithm is combined to predict the development time. If there are configuration problems, the system returns to S2 for adjustment and replanning. S4. Development process control and quality inspection based on Flink real-time stream computing: Develop according to the S3 plan, collect and analyze process data through the Flink framework, detect code quality by combining the AST algorithm, and carry out quality inspection by the defect density quantization algorithm. If there are any abnormalities or the inspection fails, return to the corresponding previous steps for rectification. S5. Risk warning and handling based on DNN deep learning model: Throughout the process from S2 to S4, the DNN model is used to monitor and predict risks, and the Apriori algorithm is used to match historical handling solutions. If the risk is not eliminated, the corresponding step is returned for adjustment. S6. Project acceptance and data review based on data mining algorithms: Acceptance is conducted according to standards. If it fails, it is returned to S4 for rectification. If it passes, data patterns are mined based on the FP-Growth algorithm, and the database and parameters of each intelligent model in S1 are updated in combination with the Bayesian optimization algorithm.

2. The software development management method based on big data according to claim 1, characterized in that: In S1, the multi-source real-time acquisition terminal includes an IoT acquisition terminal, a development tool plug-in terminal, and a system docking terminal. The acquired data covers demand data, personnel data, resource data, development process data, quality inspection data, and environmental data. During the data collection process, a unique identifier algorithm is used to uniformly record the collection time, subject, related project information, and unique identifier ID of the data source for the raw data. The data is then transmitted with low latency via a Kafka message queue.

3. The software development management method based on big data according to claim 1, characterized in that: In S1, the data preprocessing desensitization process adopts the national cryptographic SM4 symmetric encryption algorithm. The preprocessing process records operation logs through the ELK distributed log framework. The preprocessing effect is quantitatively verified based on the Spark distributed computing framework. The preset data integrity qualification threshold is 95%, and the coefficient of variation qualification threshold is 0.

2. If the standards are not met, the process is returned to the corresponding stage for reprocessing through Flink stream computing tasks. Each data partition of the distributed database is marked with a unique collection source identifier through a hash algorithm.

4. The software development management method based on big data according to claim 1, characterized in that: In S2, the GBDT machine learning model is trained based on the TensorFlow framework and its accuracy is optimized using cross-validation. The model is input with technical, cost, and cycle indicators as feature values ​​to obtain a quantitative score. The logic conflict of sub-requirements is identified by a graph structure conflict detection algorithm. The feasibility of the requirements is graded according to the preset scoring criteria. If the requirements are not qualified or there is a conflict, the system returns to S1 to supplement data and re-analyze.

5. The software development management method based on big data according to claim 1, characterized in that: In S3, when formulating a project planning scheme, the historical project, personnel and resource data in S1 are combined to clarify the development cycle plan, milestone nodes, deliverable requirements and responsible parties. Personnel allocation uses the personnel load balance as the fitness function, and resource scheduling is completed based on a greedy algorithm combined with dynamic programming. The usage time, quota, and maintenance responsibility of each resource are clearly defined, and the rationality of the plan is verified through quantitative indicators.

6. The software development management method based on big data according to claim 1, characterized in that: In S4, the development process control will conduct real-time correlation and comparison analysis of the collected data with the historical data of S1 and the planning data of S3. The AST algorithm will identify code vulnerabilities, redundancies and non-standard content through syntax tree traversal. After the quality inspection is completed, the results, defect information and handling status will be recorded to form a quality inspection report. The features of high-quality code will be extracted by data mining algorithms and incorporated into the S1 experience database.

7. The software development management method based on big data according to claim 1, characterized in that: In step S4, if the progress deviation exceeds a preset threshold, the process returns to step S3 to adjust the planning and resource allocation scheme through an intelligent scheduling algorithm. When a quality inspection fails, the root cause analysis algorithm based on fishbone diagrams and association rules is used to identify the cause and implement corrective measures. If a major technical problem occurs and the process cannot proceed, return to S2 to adjust the demand breakdown plan or return to S1 to supplement data and restart the process.

8. The software development management method based on big data according to claim 1, characterized in that: In S5, the DNN deep learning risk warning model is built on the PyTorch framework and uses stochastic gradient descent (SGD) optimization. It takes data on demand changes, schedules, quality defects, resources, and personnel as input features, clarifies the warning indicators and thresholds for five types of risks, identifies potential risks and classifies them into levels through model feature extraction and nonlinear mapping, and executes the processing plan after secondary verification by the expert system. If the standard is not met, the model parameters are optimized through transfer learning algorithm.

9. The software development management method based on big data according to claim 1, characterized in that: In S6, the data review extracts the experience, problems, risk handling, and acceptance data of the entire process. After feature extraction and pattern mining, it is added to the distributed database in S1. The parameters and rules of each intelligent model are adjusted through grid search and Bayesian optimization algorithms. The review results, model update records, and adjustment plans are stored in the database through data serialization algorithms.

10. A software development management method based on big data according to claim 1, characterized in that: In step S1, a dedicated big data database is established to store the standardized datasets that have been preprocessed in this step. The standardized datasets are stored in partitions according to data type, and each data partition is marked with a unique source identifier.

Images