Adaptive source code quality assessment

Abstract

An adaptive quality assessor assesses quality of source code, taking into account feedback from software engineers. A modification to source code of a software system is detected. A plurality of metrics are selected from source code metrics available for the source code, the selection being at least in response to feedback about the modification to the source code. A score is computed comprising a weighted combination of the selected plurality of metrics. An automated action on a consuming process is triggered in response to the computed score, such as to control integration of the modified source code into the software system or to prioritize tasks in a queue.

Description

ADAPTIVE SOURCE CODE QUALITY ASSESSMENT

[0001] The present disclosure relates to an adaptive quality assessor such as for determining the quality of software using adjustable benchmarks. The disclosure is particularly related to, but not limited to, control of quality gates admitting source code to a source code repository.BACKGROUND

[0002] Standardised software quality systems aim to monitor software quality through standardised software quality measurements in order to keep the quality to a high standard.

[0003] There is a growing desire to use standardised software quality systems in many fields, including but not limited to telecommunications and communications network services however this presents a number of challenges. One of these challenges is that using a standardised software quality system will not necessarily meet current requirements for monitoring software quality.

[0004] The examples described herein are not limited to examples which solve problems mentioned in this background section.SUMMARY

[0005] Examples of preferred aspects and embodiments of the invention are as set out in the accompanying independent and dependent claims.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] In an aspect of the technology there is a computer implemented method comprising: detecting a modification to source code of a software system; selecting a plurality of metrics from source code metrics available for the source code, the selection being at least in response to feedback about the modification to the source code; computing a score comprising a weighted combination of the selected plurality of metrics; triggering an automated action on a consuming process in response to the computed score.

[0008] Because the selection is made in response to feedback about the modification to the source code, such as feedback from software engineers in a team working on the source code, the score is tailored to the particular use case and optionally to the particular domain of the software system. By triggering the automated action the score is used to efficiently operate the consuming process such as to control integration of the modification of the source code to the software system or for other consuming processes.

[0009] Quality metrics selected from a predefined software house metrics pool may be used and there is ability to update the importance of each metric in the selected suite in response to team skills as well as observed software quality through a continuous feedback loop. This is achieved by assigning weights, which may be configurable, for each selected metric and continuous monitoring of software quality as well as feedback such as team lead input to adjust metric weights accordingly.

[0010] The technology is particularly useful with auto-generated software components that interact or interface with human-developed software, such as contemporaneously developed software or pre-existing legacy software. Feedback can be given so that quality metrics which are selected and weights which are selected are different for human generated source code than for auto generated source code. This is found particularly useful as characteristics of human generated source code may be different from auto generated source code.

[0011] Preferably the selection takes into account historical quality data about the source code. By taking into account historical quality data such as historical values of the source code metrics it is possible to select combinations of metrics which are likely to be reliable indicators of quality of the source code.

[0012] Preferably the historical quality data comprises historical values of the source code metrics and a statistic representing the historical values, such as an average or the result of a limit operation on the historical values. In some cases the historical quality data is represented in graphical form, wherein the graphical form includes, but is not limited to, line charts, area charts, bar charts, candlestick charts, scatter plots, box plots, histograms, and any other type of graphing method capable of presenting data over time. Using a graphical representation of the historical values, or a statistic computed from the historical values, is found to enable reliable indication of quality of the source code even where the metrics themselves are not direct indicators of quality.

[0013] The computer implemented method as claimed in any preceding claim wherein the source code metrics are computed from the source code using one or more automated metric collection tools. This gives an efficient reliable way to compute values of the metrics from the source code.

[0014] In various examples the method comprises selecting values of the weights at least in response to the feedback and optionally also in response to historical quality data about the source code. In this way the weighted combination is adapted according to the feedback.

[0015] In various examples the selection is computed using any of: a machine learning model, rules, an optimization process. These are found to be efficient ways of computing the selection that work well in practice.

[0016] In various examples the method comprises comparing the score with the feedback and in response to the comparison updating the selection of the plurality of metrics and weights of the weighted combination. In this way the quality assessor is adaptable.

[0017] In some examples the consuming process is any of: an automated process that prioritizes tasks for modifying the source code or an automated continuous integration quality gate for controlling integration of the modification of the source code into the software system. Triggering the consuming process which is automated gives reliability, reduces errors, and allows for the historical data and feedback to be taken into account which otherwise is impractical to achieve.

[0018] In various examples the method comprises receiving the feedback as user input from one or more software engineers. The feedback may be requested by displaying a request in a software development environment being used to modify the source code. This enables feedback to be obtained in an efficient manner and improves the accuracy of the feedback. Thus the adaptive quality assessor is able to use universally understood metrics, and also may be tuned and adjusted to team project specifics.

[0019] According to another aspect of the technology there is an adaptive quality assessor comprising: a processor; a memory, the memory storing instructions which when executed by the processor implement the method of: detecting a modification to source code of a software system; selecting a plurality of metrics from source code metrics available for the source code, the selection being at least in response to feedback about the modification to the source code; computing a score comprising a weighted combination of the selected plurality of metrics; triggering an automated action on a consuming process in response to the computed score.

[0020] According to another aspect of the technology there is a computer program comprising instructions which when executed on a processor implement the method of any of the examples described herein.

[0021] It will also be apparent to anyone of ordinary skill in the art, that some of the preferred features indicated above as preferable in the context of one of the aspects of the disclosed technology indicated may replace one or more preferred features of other ones of the preferred aspects of the disclosed technology. Such apparent combinations are not explicitly listed above under each such possible additional aspect for the sake of conciseness.

[0022] Other examples will become apparent from the following detailed description, which, when taken in conjunction with the drawings, illustrate by way of example the principles of the disclosed technology.BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG. 1 is a schematic diagram of an adaptive quality assessor for use with software systems;

[0024] FIG. 2 is a schematic diagram of an adaptive quality assessor, such as that of FIG. 1 , in more detail;

[0025] FIG. 3 is a flow diagram of a method performed by an adaptive quality assessor;

[0026] FIG. 4 is a schematic diagram of a computing-based device for deploying an adaptive quality assessor such as that of FIG. 1 or FIG. 2.

[0027] The accompanying drawings illustrate various examples. The skilled person will appreciate that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the drawings represent one example of the boundaries. It may be that in some examples, one element may be designed as multiple elements or that multiple elements may be designed as one element. Common reference numerals are used throughout the figures, where appropriate, to indicate similar features.DETAILED DESCRIPTION

[0028] The following description is made for the purpose of illustrating the general principles of the present technology and is not meant to limit the inventive concepts claimed herein. As will be apparent to anyone of ordinary skill in the art, one or more or all of the particular features described herein in the context of one embodiment are also present in some other embodiment(s) and / or can be used in combination with other described features in various possible combinations and permutations in some other embodiment(s).

[0029] One of the main concerns for any software house is how good the software and applications they develop are. For these reasons, many software houses stick with standardised software quality measurements to monitor the software quality and ensure keeping the quality to a high standard. However, using a standardised software quality system will not necessarily meet all software companies’ requirements for monitoring software quality. Companies that are involved in a range of software projects generally will have overlapping software teams that can also incidentally be augmented with subcontractors who may be assigned to different software developers’ teams. While some aspects of software quality apply universally, a large number of software quality aspects correlate with the skills, vision and ways of working of the software teams themselves. The inventors have recognized that what is needed, therefore, is a software quality assessment tool consisting of universally understood metrics, but which can be tuned and adjusted to team project specifics.

[0030] A challenge is that each software team working on source code of a particular software system might work differently and therefore a quality benchmark might need tovary between teams. Additionally, different technologies or patterns used within the same team might not be comparable with standardised quality measurement benchmarks and might therefore need custom benchmarks.

[0031] In addition to providing core functionality, it is of equal importance for software systems to have the right quality attributes. These quality attributes can cover a wide range of aspects, ranging from speed and stability to ease of maintenance and evolvability. Achieving those qualities requires in-depth understanding of the source code, and this insight is typically attained by measuring various aspects of the source and architecture. Metrics used for these kinds of insights include number of lines of code, depth of inheritance, cyclomatic complexity, etc.

[0032] While such metrics in their own right give no direct insight into the quality attributes of the software system, their progression over time can provide a valuable insight into pain points and areas that require improvement. Software development therefore systematically collect metrics and graphs them over time to help inform tasks to be done and their prioritisation.

[0033] Systematic metric collection during software development is supported by various tools that integrate into existing software toolchains, such as Sonarqube (trade mark) and Structure 101 (trade mark). This means metric values are collected automatically and plotted over time to provide software development teams easy access. However, specialising metric collection and interpretation towards team-specific metrics is not supported. The inventors have also recognised that there are no possibilities for feedback regarding the metrics.

[0034] The present technology gives the ability to specify quality metrics selected from a predefined metrics pool and the ability to update the relative importance of each metric in the selected suite in response to team skills as well as observed software quality through a continuous feedback loop. This is achieved by assigning configurable weights for each selected metric and developing a process to automatically adjust the metric weights.

[0035] In various examples there is a software quality metric suite that can adjust to team skills as well as currently observable software quality and risk insight.

[0036] Such software quality measurements are useful to be consumed by automated processes, such as to help prioritize source code in a queue waiting to be upgraded or maintained, or to control what source code is admitted into a source safe or repository holding production software.

[0037] FIG. 1 is a schematic diagram of an adaptive quality assessor 102 for use with a software system 200. The software system 200 is for any purpose, including but not limited to: managing a communications network 104, implementing a voicemail service, encrypting and decrypting packets on a communications link. It is desired to monitorquality of the software system 200 in an adaptive manner as changes are made to the software system 200

[0038] The adaptive quality assessor 102 is computer implemented and in the example of FIG. 1 is deployed as a cloud service such that the source code 100 and optionally the software system 200 are accessible to the adaptive quality assessor 102. Communications network 104 may be the internet or any other communications network. However, deployment as a cloud service is not essential as the adaptive quality assessor 102 is also deployed at an on premises server or in any computer with access to source code 100 of the software system 200 in some examples. The source code 100 of the software system 106 may be stored in a source code repository 216 in some examples. In some examples the adaptive quality assessor 102 controls gates to the source code repository 216 in an automated manner responsive to quality measurements. In some examples the adaptive quality assessor 102 provides quality scores to consuming processes 216. The quality scores trigger automated consuming processes such as prioritization 218 of tasks concerning the source code 100 in a queue of tasks and / or automated continuous integration quality gates 220 controlling what source code is admitted to the source code repository 216.

[0039] FIG. 2 is a schematic diagram of an adaptive quality assessor 102, such as that of FIG. 1, in more detail. The adaptive quality assessor 102 comprises a quality scoring module 208, a score 202, a feedback collector 204 and a review module 206. The adaptive quality assessor 102 computes output comprising a numerical quality score which is sent to one or more consuming processes 216. The adaptive quality assessor 102 receives input from a software system 200. The input comprises source code of the software system and optionally version management information about the source code. The source code is fitted with metric collection tools such as Sonarqube, Structure 101 or others.

[0040] Within the quality scoring module is a metrics pool 210 a quality evaluator 210 a store of historical quality data 212 and a metrics and weight selector 214.

[0041] The metrics pool 210 is a store of a plurality of metrics for measuring characteristics of the software system 200. A non-exhaustive list of examples of metrics is: number of identified bugs, number of lines of source code, depth of inheritance, cyclomatic complexity, speed of execution, test coverage, number of security issues.

[0042] The store of historical quality data 212 comprises values of the metrics received in the past. The historical quality data 212 may comprise a plurality of graphs, one per metric, each graph plotting values of one of the metrics which have been measured from the software system 200 over time. While the metrics in their own right may not give direct insight into quality attributes of the software system (such as ease of maintenance,evolvability, stability), their progression over time can provide a valuable insight into quality attributes of the software system.

[0043] The metrics and weight selector 214 is functionality to select which metrics are to be used by the quality evaluator 210 and what weights are to be applied to those metrics by the quality evaluator 210. The quality evaluator 210 uses a scoring function comprising a weighted combination of two or more scoring metrics. The metrics and weight selector 214 selects one or more metrics from the metrics pool 210. The metrics and weight selector 214 selects metrics and sets values of the weights using one or more rules. When selecting the metrics and setting the values of the weights, the metrics and weight selector 214 takes into account one or more of: the historical quality data 212, skills of a team of software engineers working on the source code, the technologies used in the software system under development.

[0044] The quality evaluator 210 computes a quality score by computing a weighted combination of current values of the selected metrics. The weights are the ones selected by the metrics and weight selector 214. The current values of the selected metrics are available to the quality scoring module 208 because they are input from the metric collection tools fitted to the software system 200. The quality score, once computed, is stored in the historical quality data 212. The quality score is sent to score component 202 within the adaptive quality assessor 102. In some cases, the quality evaluator 210 also outputs indications of elements of the source code which have a quality score below a threshold.

[0045] A feedback collector 204 of the adaptive quality assessor 102 acts to request and record feedback from software engineers, users, or other computing systems, about the software system 200. When a merge request is received at a source code repository of the software system, the feedback collector 204 sends a request to software engineers in a team who are working on the maintenance or development of the software system. The feedback request asks the software engineers for information about quality of the updated software system, updated as a result of the merge request. Feedback is received from the software engineers in the form or a quality score or category rating. The feedback is stored and sent to the review module 206.

[0046] The review module receives as input feedback from the feedback collector 204 and the score. The review module compares the feedback and the score, such as by subtracting the score from the feedback (where the feedback is a numerical value on the same scale as the score). If the difference between the feedback and the score is above a threshold the review module sends information about the difference to the metrics and weight selector 214. The metrics and weight selector 214 is then able to use the information about the difference to select metrics and weights in order to try to minimize the difference between the feedback and the score.

[0047] The output from the adaptive quality assessor 102 is sent to one or more consuming processes 216 and triggers actions at one or more of the consuming processes 216. FIG. 2 shows two examples of consuming processes 216 but is not limiting as there are other consuming processes in other examples. One of the consuming processes 216 shown in FIG. 2 is an automated backlog prioritisation 218 comprising a queue. This processes orders items in the queue in response to the output from the adaptive quality assessor 102. In an example the items in the queue are tasks that software developers may complete to improve the software system.

[0001] Another of the consuming processes 216 shown in FIG. 2 is an automated continuous integration quality gates process 220. This process controls gates to a source code repository responsive to the output from the adaptive quality assessor 102. In an example, where a quality score is below a threshold the quality gates are closed and where a quality score is above the threshold the quality gates are open admitting the changes to the source code into the source code repository. The software quality advice is usable to protect against regressions in the form of automated quality gates in a continuous integration process. In some examples the consuming processes comprise automated instrumentation of development processes linking into tooling such as GitLab and Jenkins Pipelines as well as deployment orchestration tools.

[0002] FIG. 3 is a flow diagram of a method performed by an adaptive quality assessor 102 such as that of FIG. 1 or FIG. 2.

[0003] The adaptive quality assessor accesses 300 source code of a software system. The source code is version managed and fitted 302 with metric collection tools such as Sonarqube or Structure 101 or others.

[0004] The adaptive quality assessor selects 304 a plurality of metrics from a metrics pool, which are the metrics available from the metric collection tools. The adaptive quality assessor assigns weights to the selected metrics for use in making a weighted combination of values of the selected metrics. To make the selection the adaptive quality assessor uses rules and historical quality data to make the selection. In situations where there is not yet any historical quality data available the selection is made at random or using a default setting configured by an operator.

[0005] The adaptive quality assessor waits and checks 306 for a modification to the source code. A modification made to the software system triggers a quality assessment 308 by the quality evaluator 210 based on the currently selected weighted metrics (from operation 304), which results in a software quality score. The software quality score is used to trigger 310 one or more downstream processes in response to the score as explained with reference to FIG. 2.

[0006] The adaptive quality assessor collects 312 feedback from software developers. Developer feedback is collected on the changes made to the source code that weredetected in operation 306, how much effort was involved and how quality has changed over time.

[0007] The score computed at operation 308 is compared 314 with the feedback. The feedback is cross-referenced with the quality score by the review module 206 to determine accuracy, and if needed a recommendation is made to the metrics and weight selector 214 on how the quality score should be adjusted 316.

[0008] The metrics & weight selector takes into account the recommendation and together with the historical data a new weighted metric suite is selected that will be used for future assessments by the quality evaluator 210. The metrics and weight selection performed is performed using rules or techniques from machine learning or optimisation theory, for which mature libraries and processes exist.

[0009] In an example the metrics and weight selector 214 comprises a machine learning model which has been trained using reinforcement learning. The machine learning model comprises a neural network policy function which predicts a next action from a plurality of possible next actions which are the possible combinations of the metrics and weights. When the predicted action is taken (i.e. the quality evaluator 210 uses the selected weighted metrics) the effects in the environment are observed (i.e. the score 202 and the feedback from feedback collector 204). The effects in the environment are used to update the policy function (the neural network) using any suitable reinforcement learning objective function.

[0010] Note that the order of the operations in FIG. 3 is changed in some examples and some of the operations may be carried out in parallel. The collection of the feedback 312 may be performed in parallel with the quality assessment 308.

[0011] FIG. 4 illustrates various components of an example computing device 400 in which examples of an adaptive quality assessor 102 are implemented. The computing device is of any suitable form such as a compute node in a data centre, an on premises server, a desktop computer.

[0012] The computing device 400 comprises one or more processors 402 which are microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to perform the methods of figures 1 to 3 such as for assessing quality of source code in an adaptable manner. In some examples, for example where a system on a chip architecture is used, the processors 402 include one or more fixed function blocks (also referred to as accelerators) which implement a part of the method of figures 1 to 3 in hardware (rather than software or firmware). That is, the methods described herein are implemented in any one or more of software, firmware, hardware. The computing device has a data store 406 holding feedback from software engineers, scores, metrics, weights. The computing device has an adaptive quality assessor 102 in memory 414. Platform softwarecomprising an operating system 416 or any other suitable platform software is provided at the computing-based device to enable application software 418 to be executed on the device. Although the computer storage media (memory 414) is shown within the computing-based device 400 it will be appreciated that the storage is, in some examples, distributed or located remotely and accessed via a network or other communication link (e.g. using communication interface 404).

[0013] The computing-based device 400 also comprises an input / output controller 408 arranged to output display information to a display device 410 which may be separate from or integral to the computing-based device 400. The display information may provide a graphical user interface to enable software engineers to input feedback and to view scores. The input / output controller 408 is also arranged to receive and process input from one or more devices, such as a user input device 412 (e.g. a mouse, keyboard, camera, microphone or other sensor). In some examples the user input device 412 detects voice input, user gestures or other user actions. In an example the display device 410 also acts as the user input device 412 if it is a touch sensitive display device. The input / output controller 408 outputs data to devices other than the display device in some examples.

[0014] Any reference to 'an' item refers to one or more of those items. The term 'comprising' is used herein to mean including the method blocks or elements identified, but that such blocks or elements do not comprise an exclusive list and an apparatus may contain additional blocks or elements and a method may contain additional operations or elements. Furthermore, the blocks, elements and operations are themselves not impliedly closed.

[0015] The steps of the methods described herein may be carried out in any suitable order, or simultaneously where appropriate. The arrows between boxes in the figures show one example sequence of method steps but are not intended to exclude other sequences or the performance of multiple steps in parallel. Additionally, individual blocks may be deleted from any of the methods without departing from the spirit and scope of the subject matter described herein. Aspects of any of the examples described above may be combined with aspects of any of the other examples described to form further examples without losing the effect sought. Where elements of the figures are shown connected by arrows, it will be appreciated that these arrows show just one example flow of communications (including data and control messages) between elements. The flow between elements may be in either direction or in both directions.

[0016] Where the description has explicitly disclosed in isolation some individual features, any apparent combination of two or more such features is considered also to be disclosed, to the extent that such features or combinations are apparent and capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features orcombinations of features solve any problems disclosed herein. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A computer implemented method comprising detecting a modification to source code of a software system; selecting a plurality of metrics from source code metrics available for the source code, the selection being at least in response to feedback about the modification to the source code; computing a score comprising a weighted combination of the selected plurality of metrics; triggering an automated action on a consuming process in response to the computed score.

2. The computer implemented method as claimed in claim 1 , wherein the selection takes into account historical quality data about the source code.

3. The computer implemented method as claimed in claim 2, wherein the historical quality data comprises historical values of the source code metrics.

4. The computer implemented method as claimed in any preceding claim wherein the source code metrics are computed from the source code using one or more automated metric collection tools.

5. The computer implemented method of any preceding claim comprising selecting values of the weights at least in response to the feedback.

6. The computer implemented method of any preceding claim comprising selecting values of the weights at least in response to historical quality data about the source code.

7. The computer implemented method of any preceding claim wherein the selecting is computed using any of: a machine learning model, rules, an optimization process.

8. The computer implemented method of any preceding claim comprising comparing the score with the feedback and in response to the comparison updating the selection of the plurality of metrics and weights of the weighted combination.

9. The computer implemented method of any preceding claim wherein the consuming process is any of: an automated task that prioritizes tasks for modifying the source code, an automated continuous integration quality gate for controlling integration of the modification of the source code into the software system.

10. The computer implemented method as claimed in any preceding claim comprising receiving the feedback as user input from one or more software engineers.

11. The computer implemented method as claimed in claim 2, wherein the historical quality data comprises historical values of the source code metrics and a statistic representing the historical values.

12. The computer implemented method of any preceding claim wherein selection of the metrics and the weight is computed using a reinforcement learning model.

13. The computer implemented method of any preceding claim comprising requesting the feedback by displaying a request in a software development environment being used to modify the source code.

14. An adaptive quality assessor comprising: a processor; a memory, the memory storing instructions which when executed by the processor implement the method of: detecting a modification to source code of a software system; selecting a plurality of metrics from source code metrics available for the source code, the selection being at least in response to feedback about the modification to the source code; computing a score comprising a weighted combination of the selected plurality of metrics; triggering an automated action on a consuming process in response to the computed score.

15. A computer program comprising instructions which when executed on a processor implement the method of any of claims 1 to 13.

Images