Software engineering source code segmentation method and system

Abstract

The invention discloses a software engineering source code segmentation method and system, and relates to the technical field of software engineering. Comprising the following steps: S1, segmenting a source code file by using a deep machine learning method; before formal segmentation operation is carried out, code file samples which are in the current market and have the same programming language as a source code file needing segmentation are collected to serve as a data training set to carry out data learning training, code fragments are manually marked and classified before training, and the code fragments are classified into different code blocks; segmenting the source code through deep mechanical learning and obtaining a plurality of software modules; the source code file is automatically segmented by using the deep machine learning method, the efficiency is higher, more complex modes and dependency relationships can be identified, dependence on manual intervention is reduced, the segmentation efficiency is improved, manual review is performed after testing, errors caused by automatic segmentation are avoided, and the method is suitable for popularization and application. And the reliability and the running speed of the software are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software engineering, and in particular to a method and system for segmenting software engineering source code. Background Art

[0002] Software engineering is a systematic and standardized software development process that aims to improve the quality, efficiency and reliability of software development. It includes a series of activities from requirements analysis, design, coding, testing, deployment to maintenance, emphasizing methods such as teamwork, documentation, testing and iterative development. The goal of software engineering is to produce products with correctness, usability and reasonable cost, ensure that the software meets user needs, and effectively manage software development projects; the source code files of software engineering projects are generally large and slow to load and use, so it is necessary to segment the source code to optimize application performance. By segmenting the code, the huge code base can be broken down into small, manageable parts, which can be loaded separately on demand. This method not only improves the loading speed of the first screen, but also allows more content to be dynamically loaded when the user interacts with the application, realizing the so-called "lazy loading". Specifically, source code segmentation can effectively reduce the size of requested resources, speed up system response speed, and optimize user experience. At the same time, it separates third-party dependent libraries from business code, so that even if the business code changes, as long as the version or code of the dependent library is not changed, the cached library files are still valid, further improving the response speed.

[0003] After searching, the Chinese patent publication number CN101826014B discloses a method for dividing the source code of a software project. After dividing the source code of a software project, two new software projects are generated: the first is a link library file generated by extracting part of the reusable source code from the software project, which can be reused in other software projects to save software development costs; the second is a new software project obtained by merging the remaining source code after the software project is divided with the generated link library file. The above patent has the problem that the division needs to be carried out step by step. When processing large software projects, the source code needs to be divided into a large number of sub-components, and the efficiency of step-by-step division is very low. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a method and system for segmenting software engineering source code.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for segmenting software engineering source code comprises the following steps:

[0007] S1: Use deep machine learning methods to segment source code files. Before the formal segmentation operation, first collect code file samples in the same programming language as the source code files that need to be segmented on the market as data training sets for data learning training. Before training, manually mark and classify these code snippets and classify them into different code blocks. Through deep machine learning, the source code is segmented and multiple software modules and software components are obtained.

[0008] S2: Summarize the directories of the software modules and software components separated by the code; use Webpack and React to build, realize dynamic loading, and use Python scripts to automatically generate directory structures according to preset rules to ensure the consistency and accuracy of the code directory; manage and summarize the code directory through the Subversion version control system during the directory generation process, and record and manage code changes through folder structures and commit records;

[0009] S3: Test the software modules and software components after segmentation; preferably use automated testing, write test scripts to automatically execute test tasks, test a large number of components quickly and accurately, test UI automation, interface automation and unit test automation, detect the running speed and whether errors are reported after automatic segmentation and directory aggregation, and use performance analysis tools to monitor the running status of components, CPU usage, and memory usage;

[0010] S4: Manually handle components that report errors and are too slow. First, check the previously added feature tags and source code directory structure to confirm the dependencies and interactions between multiple components, quickly locate the problem, and then perform single component testing to reproduce the problem. Collect key information such as error manifestations and trigger conditions, and observe program state changes by single-step code execution to find potential sources of errors. Analyze log files to check system logs or error logs to find error stacks. Next, check whether the component interface definition is normal after source code segmentation to prevent errors caused by communication failures between components due to incorrect interface definitions. At the same time, check the interface definition document to ensure that the data types, parameters, and return values ​​passed between interfaces are correct. For components that are too slow, the running speed can be improved by optimizing the code to eliminate logical errors and non-compliant coding caused by automatic segmentation.

[0011] Preferably: the deep machine learning model in step S1 adopts the Transformer architecture.

[0012] Further: In the step S1, the Transformer model is used in conjunction with TransFuse to first convert the data training set format into a sequence format suitable for Transformer processing. At this time, each character or word in the source code is regarded as an element, and these words or elements are organized into a sequence, and multiple subsequences are composed of a long sequence for conversion;

[0013] The Transformer model is then used to encode the converted dataset sequence. The Transformer model captures the global dependencies in the sequence through the self-attention mechanism, thereby learning the semantic representation of the source code. The training set sequence encoded by the Transformer is then input into the TransFuse model. The TransFuse model combines the advantages of the Transformer and CNN, and can capture local features while retaining the global information of the training set sequence.

[0014] In the TransFuse model, convolutional layers are used to extract local features of the training set sequence. These local features can help distinguish different parts of the code. The features extracted by the convolutional layer are fused with the global features encoded by Transformer to obtain the final source code segmentation result.

[0015] Further: in the step S2, the dynamic import() syntax is inserted into the divided software modules and software components so that the corresponding specific module or function can be loaded when the user only uses a specific function, and the corresponding module can be dynamically imported.

[0016] As a preferred solution of the present invention: using the SplitChunksPlugin plugin in the S2 step can further optimize code segmentation, extract the same part of the code to speed up the extraction speed; and using a Python script to automatically generate a directory structure according to preset rules to ensure the consistency and accuracy of the code directory.

[0017] As a further solution of the present invention: the automated test script in the S3 step uses Apifox, and Apifox writes JavaScript scripts to test the API, creates requests and writes custom scripts, verifies the returned results, and ensures that each API functions correctly.

[0018] As a further solution of the present invention: the test results in the S3 step are recorded and summarized using Allure to generate a visual test report for checking the execution status of the components in the test, success and failure statistics, and detailed error information, so that developers can quickly find the parts of the test that execute too slowly and report errors.

[0019] Based on the above solution: In the S4 step, the SplitChunksPlugin tool of Webpack is used to manually re-split. By optimizing the option of splitChunks in the automatically split component configuration in the SplitChunksPlugin plug-in of Webpack, deduplication and separation of modules can be achieved.

[0020] Based on the previous solution: In the S4 step, Webpack is used to generate a separate bundle for each entry point. By configuring splitChunks.cacheGroups, it can be ensured that modules with the same dependencies are only included in one chunk to prevent code duplication.

[0021] Based on the above scheme: the S4 step uses Rollup as a packaging tool to implement module segmentation through its built-in code segmentation function, and imports the syntax import() in the component to load the module on demand; then Rollup is used to statically analyze the import and export statements in the code, remove unused code, and further reduce the size of the packaged file when cooperating with code segmentation.

[0022] The beneficial effects of the present invention are:

[0023] 1. A method and system for segmenting software engineering source code, which automatically segments source code files using deep machine learning methods, is more efficient, can identify more complex patterns and dependencies, reduces reliance on manual intervention, improves segmentation efficiency, and performs manual review after testing, avoiding errors in automatic segmentation and improving the reliability and running speed of the software.

[0024] 2. A method and system for segmenting software engineering source code, by inserting dynamic import() syntax into the segmented software modules and software components, so that the corresponding specific modules or functions can be loaded when the user only uses a specific function, and the corresponding modules can be dynamically imported; the system response speed can be improved, and the initial loading time of the software can be reduced, thereby improving the software response speed and user experience, and significantly improving performance under limited hardware resources.

[0025] 3. A method and system for segmenting software engineering source code, which can effectively perform fast and accurate testing on a large number of components by writing test scripts to automatically execute test tasks, saving time and resources. It can test UI automation, interface automation and unit test automation, and reduce the time consumed by human detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1The present invention provides a method and system for segmenting software engineering source code. DETAILED DESCRIPTION

[0027] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.

[0028] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0029] Embodiment 1:

[0030] A method and system for segmenting software engineering source code, such as Figure 1 As shown, the following steps are included:

[0031] S1: Use deep machine learning methods to segment source code files. Before the formal segmentation operation, it is necessary to first collect code file samples in the same programming language as the source code files that need to be segmented on the market as data training sets for data learning training, and manually mark and classify these code fragments before training, and classify them into different code blocks;

[0032] Preferably, Transformer can be used for source code segmentation. Transformer is a deep learning model with an encoding-decoding structure and a self-attention mechanism, which is used to process sequence data. TransFuse is used in conjunction with the Transformer architecture to perform code segmentation and information fusion when processing source code multi-source information.

[0033] When using it, first convert the data training set format into a sequence form suitable for Transformer processing. At this time, each character or word in the source code is regarded as an element, and these words or elements are organized into a sequence. Multiple subsequences are composed of a long sequence for conversion;

[0034] The Transformer model is then used to encode the converted dataset sequence. The Transformer model captures the global dependencies in the sequence through the self-attention mechanism, thereby learning the semantic representation of the source code. The training set sequence encoded by the Transformer is then input into the TransFuse model. The TransFuse model combines the advantages of the Transformer and CNN, and can capture local features while retaining the global information of the training set sequence.

[0035] In the TransFuse model, the convolutional layer is used to extract local features of the training set sequence. These local features can help distinguish different parts of the code, such as functions and variable declarations. The features extracted by the convolutional layer are fused with the global features encoded by the Transformer to obtain the final source code segmentation results. After multiple trainings, the trained Transformer model can be used to segment the source code to be segmented, and multiple software modules and software components can be obtained. A software module is a set of consistent and closely related software organizations, including programs and data structures. A software component is a self-contained, programmable, reusable, language-independent software unit suitable for assembling applications.

[0036] S2: Summarize the directories of the software modules and software components generated by code segmentation, and use Webpack and React to build them, which can realize dynamic loading and improve the overall software performance. The dynamic import() syntax is inserted into the segmented software modules and software components, so that the corresponding specific modules or functions can be loaded when the user only uses a specific function, and the corresponding modules can be dynamically imported; improve the system response speed, and reduce the initial loading time of the software, improve the software response speed and user experience, and significantly improve the performance under limited hardware resources. At the same time, the SplitChunksPlugin plug-in can be used to further optimize code segmentation, extract the same part of the code to speed up the extraction speed; and use Python scripts to automatically generate directory structures according to preset rules to ensure the consistency and accuracy of the code directory; in the process of generating directories, the code directory is managed and summarized through the Subversion version control system, and code changes can be recorded and managed through folder structures and submission records;

[0037] S3: Test the software modules and software components after segmentation. It is preferred to use automated testing. By writing test scripts to automatically execute test tasks, a large number of components can be effectively tested quickly and accurately, saving time and resources. UI automation, interface automation and unit test automation can be tested to detect the running speed and whether errors are reported after automatic segmentation and directory aggregation. The automated test script uses Apifox, and JavaScript scripts can be written to test the API, create requests and write custom scripts, verify the returned results, ensure the correct function of each API, and use performance analysis tools to monitor the running status of the components, CPU usage, and memory usage;

[0038] The test results are recorded and summarized using Allure to generate a visual test report, which is used to check the execution status of the components in the test, the success and failure statistics, and the detailed error information, so that developers can quickly find the parts of the test that are too slow to execute and report errors;

[0039] S4: Manually handle components that report errors or are too slow. First, check the previously added feature tags and source code directory structure to confirm the dependencies and interactions between multiple components, quickly locate the problem, and then perform single component testing to reproduce the problem. Collect key information such as error manifestations and trigger conditions. Observe program state changes by single-step code execution to find potential sources of errors. Analyze log files to check system logs or error logs to find error stacks. Next, check whether the component interface definition after source code segmentation is normal to prevent errors caused by communication failures between components due to incorrect interface definitions. At the same time, check the interface definition document to ensure that the data types, parameters, and return values ​​passed between interfaces are correct.

[0040] For components that are too slow, you can improve their running speed by optimizing the code, eliminating logical errors caused by automatic segmentation and non-compliant coding;

[0041] For some components with serious errors, they can be re-split through manual modification. Preferably, the SplitChunksPlugin tool of Webpack can be used for re-split. By using the optimization.splitChunks option in the automatically split component configuration in the SplitChunksPlugin plug-in of Webpack, the modules can be deduplicated and separated. Then, by setting multiple entry points, Webpack can generate a separate bundle for each entry point. By configuring splitChunks.cacheGroups, it can be ensured that modules with the same dependencies are only included in one chunk to prevent code duplication. Finally, Rollup is used as a packaging tool to implement module segmentation through its built-in code segmentation function, and the import syntax import() is used in the component to load the module on demand. Rollup is then used to statically analyze the import and export statements in the code, remove unused code, and further reduce the size of the packaged file when used in conjunction with code segmentation.

[0042] When used, this embodiment uses a deep machine learning method to automatically segment the source code file, which is more efficient and can use a large amount of data training sets to perform deep machine learning of the same language program, and then automatically extract features in the source code to achieve more accurate code segmentation. It does not rely on manually set rules and parameters in traditional segmentation methods, and automatically extracts and learns key information in the code through a multi-level network structure, so that more complex patterns and dependencies can be identified, reducing dependence on manual intervention and improving segmentation efficiency.

[0043] After training the deep learning data multiple times, the system can batch split various large-scale software source codes faster and more accurately, and quickly batch test the split software modules and software components on a large scale. After the test is completed, components with slow modification speeds and errors are manually reviewed and then re-split, ensuring that the components can run normally and quickly after the source code is split, thereby improving the reliability and running speed of the software and avoiding the problem of low efficiency and error-prone manual multiple splitting of large-scale software source code, which slowly affects the development progress in the testing and modification stages.

[0044] The above is a preferred specific implementation manner of the present invention, and the protection scope of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by any technician familiar with the field within the technical scope disclosed by the present invention in combination with the prior art or public common sense, within the spirit and principle of the present invention, shall be covered by the protection scope of the present invention.

Claims

1. A method for segmenting software engineering source code, comprising the following steps: S1: Use deep machine learning methods to segment source code files. Before the formal segmentation operation, first collect code file samples in the same programming language as the source code files that need to be segmented on the market as data training sets for data learning training. Before training, manually mark and classify these code snippets and classify them into different code blocks. Through deep machine learning, the source code is segmented and multiple software modules and software components are obtained. S2: Summarize the directories of the software modules and software components separated by the code; use Webpack and React to build, realize dynamic loading, and use Python scripts to automatically generate directory structures according to preset rules to ensure the consistency and accuracy of the code directory; manage and summarize the code directory through the Subversion version control system during the directory generation process, and record and manage code changes through folder structures and commit records; S3: Test the software modules and software components after segmentation; preferably use automated testing, write test scripts to automatically execute test tasks, test a large number of components quickly and accurately, test UI automation, interface automation and unit test automation, detect the running speed and whether errors are reported after automatic segmentation and directory aggregation, and use performance analysis tools to monitor the running status of components, CPU usage, and memory usage; S4: Manually handle components that report errors and are too slow. First, check the previously added feature tags and source code directory structure to confirm the dependencies and interactions between multiple components, quickly locate the problem, and then perform single component testing to reproduce the problem. Collect key information such as error manifestations and trigger conditions, and observe program state changes by single-step code execution to find potential sources of errors. Analyze log files to check system logs or error logs to find error stacks. Next, check whether the component interface definition is normal after source code segmentation to prevent errors caused by communication failures between components due to incorrect interface definitions. At the same time, check the interface definition document to ensure that the data types, parameters, and return values ​​passed between interfaces are correct. For components that are too slow, the running speed can be improved by optimizing the code to eliminate logical errors and non-compliant coding caused by automatic segmentation.

2. A method and system for segmenting software engineering source code according to claim 1, characterized in that: The deep machine learning model in the S1 step adopts the Transformer architecture.

3. A method and system for segmenting software engineering source code according to claim 2, characterized in that: In the step S1, the Transformer model is used in conjunction with TransFuse to first convert the data training set format into a sequence format suitable for Transformer processing. At this time, each character or word in the source code is regarded as an element, and these words or elements are organized into a sequence. Multiple subsequences are composed of a long sequence for conversion; The Transformer model is then used to encode the converted dataset sequence. The Transformer model captures the global dependencies in the sequence through the self-attention mechanism, thereby learning the semantic representation of the source code. The training set sequence encoded by the Transformer is then input into the TransFuse model. The TransFuse model combines the advantages of the Transformer and CNN, and can capture local features while retaining the global information of the training set sequence. In the TransFuse model, convolutional layers are used to extract local features of the training set sequence. These local features can help distinguish different parts of the code. The features extracted by the convolutional layer are fused with the global features encoded by Transformer to obtain the final source code segmentation result.

4. A method and system for segmenting software engineering source code according to claim 1, characterized in that: In the step S2, the dynamic import() syntax is inserted into the divided software modules and software components so that the corresponding specific module or function can be loaded when the user only uses a specific function, and the corresponding module can be dynamically imported.

5. A method and system for segmenting software engineering source code according to claim 4, characterized in that: The SplitChunksPlugin plugin is used in the S2 step to further optimize code segmentation, extract the same part of the code to speed up the extraction speed; and a Python script is used to automatically generate a directory structure according to preset rules to ensure the consistency and accuracy of the code directory.

6. A method and system for segmenting software engineering source code according to claim 1, characterized in that: The automated test script in the S3 step uses Apifox. Apifox writes JavaScript scripts to test the API, creates requests and writes custom scripts, verifies the returned results, and ensures that each API functions correctly.

7. A method and system for dividing software engineering source code according to claim 6, characterized in that: The test results in the S3 step are recorded and summarized using Allure to generate a visual test report, which is used to check the execution status of the components in the test, the success and failure statistics, and the detailed error information, so that developers can quickly find the parts of the test that are too slow to execute and report errors.

8. A method and system for segmenting software engineering source code according to claim 1, characterized in that: In the S4 step, the SplitChunksPlugin tool of Webpack is used to manually re-split. By optimizing the splitChunks option in the automatically split component configuration in the SplitChunksPlugin plug-in of Webpack, deduplication and separation of modules can be achieved.

9. A method and system for segmenting software engineering source code according to claim 8, characterized in that: In the S4 step, Webpack is used to generate a separate bundle for each entry point. By configuring splitChunks.cacheGroups, it can be ensured that modules with the same dependencies are only included in one chunk to prevent code duplication.

10. A method and system for dividing software engineering source code according to claim 9, characterized in that: The S4 step uses Rollup as a packaging tool to implement module segmentation through its built-in code segmentation function, and imports the syntax import() in the component to load the module on demand; then Rollup is used to statically analyze the import and export statements in the code, remove unused code, and further reduce the size of the packaged file when cooperating with code segmentation.

Images