Target software version evaluation method and device, electronic equipment and readable storage medium

By compiling and running the use cases of each model of the target software, dynamically adjusting the reference threshold, and calculating the version evaluation scores with the model weight, the problem that the fixed performance indicator fluctuation threshold cannot meet the needs of different model use cases is solved, and the software update quality and iterative optimization are improved efficiency.

CN118897785APending Publication Date: 2024-11-05STREAM COMPUTING INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310476805.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

In the prior art, fixed performance index fluctuation thresholds cannot effectively meet the differences in performance index distribution of different model use cases, resulting in low quality of software updates and poor results.

Method used

By compiling and running the use cases of the target software, real-time and initial performance indicator distribution intervals are obtained, differentiated comparison results are determined based on the reference threshold of the initial version, version evaluation score is calculated based on the model weight, and reference threshold is dynamically adjusted to improve accuracy.

Benefits of technology

The accuracy of the performance indicator thresholds of each model use case is improved, the quality of target software version updates is improved, and the accuracy and efficiency of software iterative optimization is ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897785A_ABST
    Figure CN118897785A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a target software version evaluation method and device, electronic equipment and a readable storage medium, and the method comprises the steps: compiling and running each model case for preset times based on different versions of target software, and obtaining a numerical value distribution interval of a corresponding target performance index, determining a reference threshold value according to an initial numerical value distribution interval corresponding to the initial version target software, and determining a version evaluation score of the current version target software according to a differential comparison result obtained by comparing a real-time numerical value distribution interval corresponding to the current version target software with the initial numerical value distribution interval, the accuracy of the reference threshold value of each model case can be improved, and the target software is iteratively optimized through the determined version evaluation score, so that the version updating quality of the target software is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a target software version evaluation method, device, electronic device and readable storage medium. Background Art

[0002] During the software development iteration process, the model use cases in the model management software will be run, and the results of the model use cases will be used to guide software updates. Taking the development iteration of compiler software as an example, the compiler performance will be evaluated by checking whether the changes in the performance indicators corresponding to the compiler running model use cases (such as the number of cycles of model inference) are within a reasonable threshold range, so that the compiler can be iteratively developed in a more optimal direction.

[0003] In the prior art, fixed fluctuation thresholds of performance indicators are usually set based on the empirical values ​​of iterative development and in combination with relevant hardware features. However, since the interval distribution of performance indicators of different models is different, the magnitude of performance indicators may even vary greatly. As the types and number of models increase, overall statistics often show that some models have increased and some have decreased, making it difficult to make a comprehensive comparison. Therefore, the use of fixed fluctuation thresholds of performance indicators obviously cannot better meet the needs of iterative development. Summary of the invention

[0004] In view of this, the embodiments of the present invention aim to provide a target software version evaluation method, device, electronic device and readable storage medium to improve the accuracy of the reference thresholds of the performance indicators of each model use case and improve the update quality of the target software version.

[0005] In a first aspect, an embodiment of the present invention aims to provide a method for evaluating a target software version, wherein the target software corresponds to a plurality of model use cases, and the method comprises:

[0006] Compile and run each of the model use cases for a round based on the current version of the target software to obtain a real-time numerical distribution interval of the target performance indicator of the corresponding model use case, wherein the round includes a preset number of times;

[0007] Compare the comparison result of the real-time numerical distribution interval with the initial numerical distribution interval corresponding to the previous round and the reference threshold of the corresponding model use case to determine the differentiated comparison result of the corresponding model use case, wherein the reference threshold is determined according to the initial numerical distribution interval corresponding to the initial version of the target software, and the reference threshold is used to characterize the allowable fluctuation degree of the target performance indicator of the corresponding model use case;

[0008] The version evaluation score of the current version of the target software is determined according to the weights corresponding to each of the model use cases and the differentiated comparison results.

[0009] Further, the reference threshold is determined based on the following steps:

[0010] Obtaining an initial value distribution interval determined after compiling and running each of the model use cases for one round based on the initial version of the target software;

[0011] Determine a relative reference value of the corresponding model use case according to the initial value distribution interval, wherein the relative reference value is used to characterize the degree of fluctuation of the target performance indicator of the corresponding model use case;

[0012] A reference threshold value corresponding to the model use case is determined according to the relative reference value.

[0013] Further, determining a reference threshold of a corresponding model use case according to the relative reference value includes:

[0014] In response to the relative reference value being not greater than a preset threshold, determining the preset threshold as a reference threshold corresponding to the model use case;

[0015] In response to the relative reference value being greater than a preset threshold, the relative reference value is determined to be a reference threshold of the corresponding model use case.

[0016] Further, determining the relative reference value of the corresponding model use case according to the initial value distribution interval includes:

[0017] Determine the interval span and average value of the target performance indicator change of the corresponding model use case according to the endpoint values ​​of the initial numerical distribution interval;

[0018] The ratio of the interval span to the average value is determined as a relative reference value for the corresponding model use case.

[0019] Furthermore, comparing the comparison result of the real-time numerical distribution interval with the initial numerical distribution interval corresponding to the previous round and the reference threshold of the corresponding model use case to determine the differentiated comparison result of the corresponding model use case includes:

[0020] Determine a real-time mean corresponding to the real-time value distribution interval and an initial mean corresponding to the initial value distribution interval according to the real-time value distribution interval and the initial value distribution interval;

[0021] Comparing the real-time mean with the initial mean to determine a rate of change of the real-time mean relative to the initial mean, wherein the rate of change is used to characterize a degree of fluctuation of a target performance indicator of a corresponding model use case;

[0022] The change rate is compared with the corresponding reference threshold to determine the differentiated comparison result of the corresponding model use case.

[0023] Furthermore, determining the version evaluation score of the current version of the target software according to the weights and differential comparison results corresponding to each of the model use cases includes:

[0024] Determine the model weight corresponding to the model use case under each model category, wherein the model weight is determined according to the preset score of each model category and the number of models of the corresponding model use case;

[0025] Based on the preset scoring rules, the intermediate scores corresponding to each model category are determined according to the differentiated comparison results of the model use cases under each model category and the model weights;

[0026] The intermediate scores corresponding to different model categories are summed to determine the version evaluation score of the target software.

[0027] Furthermore, the preset scoring rules include: the initial score of each model use case is determined based on the model weight corresponding to the model category to which it belongs, positive scoring is used for model use cases whose differentiated results have a fluctuation degree no greater than the corresponding reference threshold, negative scoring is used for model use cases whose differentiated results have a fluctuation degree greater than the corresponding reference threshold, and the sum of the initial scores of each model use case under the same category is determined as the middle score of the corresponding model category.

[0028] Furthermore, the method further comprises:

[0029] The version update status of the current version of the target software is determined according to the version evaluation score and a preset score.

[0030] Further, determining the version update status of the target software according to the version evaluation score and a preset score includes:

[0031] In response to the version evaluation score exceeding a preset score, merging the current version of the target software into a code warehouse; or

[0032] In response to the version evaluation score not exceeding a preset score, the current version of the target software is updated.

[0033] In a second aspect, an embodiment of the present invention aims to provide a target software version evaluation device, wherein the target software corresponds to a plurality of model use cases, and the device comprises:

[0034] A compiling unit, configured to compile and run a round of each of the model use cases based on the current version of the target software, and obtain a real-time numerical distribution interval of a target performance indicator of the corresponding model use case, wherein the round includes a preset number of times;

[0035] A determination unit, used to compare the comparison result of the real-time numerical distribution interval with the initial numerical distribution interval corresponding to the previous round and the reference threshold of the corresponding model use case, and determine the differentiated comparison result of the corresponding model use case, wherein the reference threshold is determined according to the initial numerical distribution interval corresponding to the initial version of the target software, and the reference threshold is used to characterize the allowable fluctuation degree of the target performance indicator of the corresponding model use case;

[0036] An evaluation unit is used to determine a version evaluation score of the current version of the target software according to the weights corresponding to each model use case and the differentiated comparison results.

[0037] In a third aspect, an embodiment of the present invention is to provide an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement any of the methods described above.

[0038] In a fourth aspect, an embodiment of the present invention aims to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps as described in any one of the above items are implemented.

[0039] The technical solution of the embodiment of the present invention compiles and runs each model use case a preset number of times based on different versions of target software to obtain the corresponding numerical distribution range of the target performance indicator, determines the reference threshold value according to the initial numerical distribution range corresponding to the initial version of the target software, and determines the version evaluation score of the current version of the target software according to the differentiated comparison result after comparing the real-time numerical distribution range corresponding to the current version of the target software with the initial numerical distribution range. This can improve the accuracy of the reference threshold value of each model use case, and at the same time, iteratively optimize the target software through the version evaluation score determined above, which is conducive to improving the update quality of the target software version. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0041] Figure 1 is a flow chart of a target software version evaluation method according to an embodiment of the present invention;

[0042] Figure 2 is a flowchart of a method for determining a reference threshold corresponding to a model use case according to an embodiment of the present invention;

[0043] Figure 3 is a flow chart of a method for determining a model use case difference comparison result according to an embodiment of the present invention;

[0044] Figure 4is a flow chart of a method for determining a version evaluation score according to an embodiment of the present invention;

[0045] Figure 5 is a flow chart of a method for determining a model use case reference threshold value according to an embodiment of the present invention;

[0046] Figure 6 is a flow chart of a method for updating target software according to an embodiment of the present invention;

[0047] Figure 7 is a schematic diagram of a target software version evaluation device according to an embodiment of the present invention;

[0048] Figure 8 is another schematic diagram of a target software version evaluation device according to an embodiment of the present invention;

[0049] Fig. 9 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, flows, components and circuits are not described in detail.

[0051] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0052] Unless the context clearly requires otherwise, words such as “include”, “including” and the like in the application documents should be interpreted as including rather than being exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0053] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0054] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0055] In the software development iteration process, the software version is generally evaluated through the running results of the software running model use case, and it is judged whether the newly added functional code of the software iteration should be merged into the code warehouse. Taking the development process of the compiler as an example, in the existing compiler development iteration process, the model use case is automatically run and the performance indicators corresponding to the running results of the model use case (such as the number of cycles of model reasoning) are used to judge whether the new code in the compiler can be merged into the code warehouse. When the fluctuation degree of the corresponding numerical value of the performance indicator of the model use case is less than the preset fluctuation threshold, it indicates that the performance indicator change of the model use case is in line with expectations, and the corresponding compiler version meets the expected requirements. The new code in the corresponding version of the compiler can be merged into the code warehouse for subsequent use.

[0056] However, in the prior art, a fixed fluctuation threshold is often set for the model use case performance indicator through existing iterative experience or in combination with the characteristics of the operating hardware environment. As the number of model use cases increases and the coverage scenarios become more and more complete, it is difficult to use this fluctuation threshold to fit the performance indicator distribution of different model use cases, resulting in low software update quality and poor effect. In view of this, an embodiment of the present invention aims to provide a target software version evaluation method to improve the accuracy of each model use case performance indicator threshold and improve the target software version update quality.

[0057] The following continues to introduce the target software evaluation method involved in the software version development process of the compiler, but it should be understood that the target software evaluation method in this embodiment can also be applied to the development iteration process of other types of software, and the relevant method steps or other contents can be adaptively adjusted according to the specific usage environment and development purpose requirements. The applicable objects and usage scenarios of this method are not limited here.

[0058] Figure 1 : is a flow chart of the target software version evaluation method of an embodiment of the present invention. Figure 1 As shown, the target software version evaluation method in this embodiment includes the following steps.

[0059] In step S100, each model use case is compiled and run for one round based on the current version of the target software to obtain a real-time numerical distribution range of the target performance indicator of the corresponding model use case.

[0060] In this embodiment, the target software is compiler software, and the target software corresponds to multiple model use cases. Each model use case is compiled and run by the compiler software, and the corresponding running results are output. The target performance indicator uses parameters related to model reasoning. Specifically, the target performance indicator in this embodiment is the number of cycles of model reasoning, which is characterized by the total number of cycles run since the NPU was reset by the counter. The smaller the number of cycles, the higher the model reasoning efficiency under the corresponding target software environment.

[0061] Optionally, one round in the present embodiment includes a preset number of times, and compiling and running one round means compiling and running each model use case for a preset number of times. Due to the influence of hardware conditions, under the same version of the target software conditions, the value of the target performance indicator corresponding to each compilation and operation of the same model may be different, but it will usually be stable in a range corresponding to a numerical distribution interval. After compiling and running each model use case for one round through the current version of the target software, the real-time numerical distribution interval of the target performance indicator of each model use case can be obtained. At the same time, the target performance indicator of the corresponding model use case is reflected by the numerical distribution interval obtained by multiple compilations and runs of each model use case corresponding to one round, which can eliminate the influence of the fluctuation of the model and the hardware environment itself on the target performance indicator in the model use case operation result to the greatest extent, so that the target performance indicator of each model use case in the current version of the target software is more accurate and more valuable.

[0062] Further, in this embodiment, the preset number of rounds corresponding to one round may be 100. It should be understood that the specific number of preset rounds may also be set and adjusted according to the actual operation of the target software, and this is not limited here.

[0063] In step S200, the comparison result of the real-time numerical distribution interval with the initial numerical distribution interval corresponding to the previous round is compared with the reference threshold of the corresponding model use case to determine the differentiated comparison result of the corresponding model use case.

[0064] In this embodiment, the reference threshold is determined based on the initial numerical distribution interval corresponding to the initial version of the target software, and the reference threshold is used to characterize the allowable fluctuation degree of the target performance indicator of the corresponding model use case. When the fluctuation degree of the target performance indicator of the model use case is less than the reference threshold, it indicates that the operation effect of the corresponding model use case under the current version of the target software is better, and the use performance of the corresponding current version of the target software is also better. On the contrary, when the fluctuation degree of the target performance indicator of the model use case is greater than the reference threshold, it indicates that the operation effect of the corresponding model use case under the current version of the target software is poor, and the target software needs to be updated to improve the use performance of the target software.

[0065] In this embodiment, the target software of the initial version used to determine the reference threshold of the model use case is the target software of the historical version. Optionally, the target software of the historical version can be the target software generated earliest in the development process of the target software. For example, assuming that the initial version of the target software is V1.0, after multiple iterations of optimization, the version of the target software changes from V1.0, V2.0, and V3.0 to the current version V4.0. When the current version V4.0 target software is evaluated, the reference threshold of each model use case in this embodiment is determined according to the initial numerical distribution interval corresponding to the target software of version V1.0. Further, when the reference threshold corresponding to each model use case is determined using the initial numerical distribution interval under the target software of the earliest generated version, once the reference threshold of each model use case is generated, it does not need to be repeatedly determined later and can be directly used. Therefore, for target software that can be applied to multiple model use cases, the use of the reference threshold determined above can save time for the subsequent version optimization of the target software, which is conducive to improving the efficiency of the target software version update.

[0066] Alternatively, the historical version of the target software used to determine the reference threshold of the model use case in this embodiment may also be the target software of the previous version closest to the current version. For example, assuming that the initial version of the target software is V1.0, after multiple iterations of optimization, the version of the target software changes from V1.0, V2.0, and V3.0 to the current version V4.0. When the current version V4.0 target software is evaluated, the reference threshold of each model use case in this embodiment is determined according to the initial data distribution interval corresponding to the target software of version V3.0. Further, when the initial numerical distribution interval of the target software of the previous version closest to the current version is used to determine the reference threshold corresponding to each model use case, before the evaluation of this target software and the iterative optimization process of the current version of the target software are started, the target version of the latest version is used to compile and run each model use case for one round, determine the initial numerical distribution interval of the corresponding target performance indicator, and determine the reference threshold of the corresponding model use case according to the initial numerical distribution interval. Therefore, compared with the reference threshold determined under the initial version of the target software, when the number of software version updates is large or fast, the reference threshold is determined by the method in this embodiment, and the reference threshold of each model use case can be dynamically adjusted according to the actual situation of the target software replacement, so that the real-time and accuracy of the reference threshold are stronger.

[0067] Figure 2 1 is a flow chart of a method for determining a reference threshold corresponding to a model use case according to an embodiment of the present invention. Figure 2 As shown, in this embodiment, the reference threshold corresponding to each model use case is determined through the following steps.

[0068] In step S210, an initial value distribution interval determined after compiling and running each model use case for one round based on the initial version of the target software is obtained.

[0069] In this embodiment, in the compilation environment corresponding to the initial version of the target software, each model use case is compiled and run for a preset number of times corresponding to a round through the initial version of the target software, and the corresponding initial value distribution interval is determined by the running results of each model use case. For example, after the model A is compiled and run by the initial version of the target software, the distribution interval of the target performance indicator cycle is [8025393.015738678, 8054098.984261322], and this interval is used as the initial value distribution interval corresponding to the model A.

[0070] In step S220, a relative reference value of the corresponding model use case is determined according to the initial numerical distribution interval, and the relative reference value is used to characterize the degree of fluctuation of the target performance indicator of the corresponding model use case.

[0071] In this embodiment, after determining the initial numerical distribution interval corresponding to the model use case, the relative reference value is determined according to the maximum and minimum values ​​in the initial numerical distribution interval. The relative reference value can be the variance of the initial numerical interval determined by the various numerical values, the maximum and minimum values ​​in the initial numerical distribution interval, or the difference between the maximum and minimum values, or other values ​​that can characterize the degree of fluctuation of the target performance indicator determined according to the maximum and minimum values ​​of the initial numerical distribution interval. Optionally, in this embodiment, the relative reference value of the corresponding model use case is determined by the following calculation formula.

[0072]

[0073] In step S230, a reference threshold corresponding to the model use case is determined according to the relative reference value.

[0074] In this embodiment, when determining the reference threshold of the corresponding model use case according to the relative reference value, the reference threshold of the corresponding model use case is determined by the relationship between the relative reference value and the preset threshold, and the preset threshold is the fluctuation degree of the target performance indicator that is acceptable by default for the existing hardware for most model use cases. Therefore, in this embodiment, different reference thresholds can be set for each model use case according to the specific usage scenario of the model use case, avoiding the situation of using a fixed reference threshold for a one-size-fits-all approach, and improving the accuracy of the reference threshold.

[0075] Optionally, in this embodiment, in response to the relative reference value being not greater than a preset threshold, the preset threshold is determined to be a reference threshold for the corresponding model use case; or, in response to the relative reference value being greater than the preset threshold, the relative reference value is determined to be a reference threshold for the corresponding model use case. That is, when the relative reference value corresponding to the model use case is less than or equal to the preset threshold, the preset threshold is determined to be the reference threshold for the model use case. When the relative reference value corresponding to the model use case is greater than the preset threshold, the relative reference value is determined to be the reference threshold for the model use case. For example, for the model use case with an initial numerical distribution interval of [8025393.015738678, 8054098.984261322], the relative reference value calculated by the above formula is 0.36. Since 0.36 is less than 0.5, the reference threshold of the corresponding model use case is determined to be 0.5. For the model use case with an initial distribution interval of [144535550.1156657, 145546646.6843343], the relative reference value calculated by the above formula is 0.7. Since 0.7 is greater than 0.5, the reference threshold of the corresponding model use case is determined to be 0.7. Therefore, for model use cases where the fluctuation of the target performance indicator is always less than or equal to the preset threshold, by determining the preset threshold as the reference threshold of the corresponding model use case, a certain fault tolerance space can be provided for the use of the model use case, the applicability of the model use case can be improved, and the update quality of the target software can be optimized; at the same time, for model use cases where the fluctuation of the target performance indicator is always greater than the preset threshold, when it is impossible to determine whether the continuous fluctuation of the performance indicator is affected by the update of the target software or the attributes of the model use case itself, by determining the relative reference value as the reference threshold of the corresponding model use case, the target performance indicator of the model use case can be made to fluctuate reasonably within a certain expected range, which is beneficial to improving the update efficiency of the target software while ensuring the update quality of the target software.

[0076] Furthermore, in this embodiment, after determining the reference threshold of each model use case, each reference threshold will be written into the configuration file as an attribute of the corresponding model use case, so that the reference threshold can be used for comparison each time the model use case is subsequently run, and the differentiated comparison of different versions of target software can be achieved based on the comparison results of each model use case under different versions, thereby evaluating the current version of the target software.

[0077] Figure 3 1 is a flow chart of a method for determining a model use case difference comparison result according to an embodiment of the present invention. Figure 3 As shown, in this embodiment, the differentiated comparison results of each model use case are determined through the following steps.

[0078] In step S310, a real-time mean corresponding to the real-time numerical distribution interval and an initial mean corresponding to the initial numerical distribution interval are determined respectively according to the real-time numerical distribution interval and the initial numerical distribution interval.

[0079] In this embodiment, since both the real-time numerical distribution interval and the initial numerical distribution interval include multiple numerical values, the real-time numerical distribution interval is represented by determining the real-time mean corresponding to the real-time numerical distribution interval, the real-time mean is determined according to the maximum and minimum numerical values ​​in the real-time numerical distribution interval, and the real-time mean is the mean of the maximum and minimum numerical values. At the same time, the initial numerical distribution interval is represented by the initial mean corresponding to the initial numerical distribution interval, the initial mean is determined according to the maximum and minimum numerical values ​​in the initial numerical distribution interval, and the initial mean is the mean of the maximum and minimum numerical values. Therefore, in this embodiment, the real-time mean corresponding to the real-time numerical distribution interval and the initial mean corresponding to the initial numerical distribution interval are determined by the above method. The determination method is simple and effective, and can provide convenience for the subsequent determination of the differentiated comparison results of each model use case according to the real-time mean and the initial mean, thereby facilitating the evaluation efficiency of the target software version.

[0080] In step S320, the real-time mean is compared with the initial mean to determine the rate of change of the real-time mean relative to the initial mean, wherein the rate of change is used to characterize the degree of fluctuation of the target performance indicator of the corresponding model use case.

[0081] In this embodiment, the real-time mean is compared with the initial mean to determine the rate of change of the real-time mean relative to the initial mean, and the rate of change of the real-time mean relative to the initial mean is used to characterize the degree of fluctuation of the target performance indicator of each model use case after being compiled and run by the current version of the target software relative to the target performance indicator after being compiled and run by the initial version of the target software.

[0082] In step S330, the change rate is compared with the corresponding reference threshold to determine the differentiated comparison result of the corresponding model use case.

[0083] In this embodiment, after determining the rate of change of the real-time mean corresponding to the current version of the target software relative to the initial mean corresponding to the initial version of the target software, the rate of change is compared with the reference threshold of the corresponding model use case, and the comparison result is determined as the differentiated comparison result of the corresponding model use case. When the rate of change is less than or equal to the reference threshold, the differentiated comparison result of the corresponding model use case is determined to be a fluctuation degree not greater than the corresponding reference threshold, indicating that the corresponding model use case has improved operating performance in the current version of the target software, and the performance of the target software has improved. When the rate of change is greater than the reference threshold, the differentiated comparison result of the corresponding model use case is determined to be a fluctuation degree greater than the corresponding reference threshold, indicating that the corresponding model use case has deteriorated operating performance in the current version of the target software, and the performance of the target software has decreased.

[0084] In step S300, the version evaluation score of the current version of the target software is determined according to the weights corresponding to each model use case and the differential comparison results.

[0085] In this embodiment, since the differential comparison results can reflect the operating effects of each model use case in the corresponding environment of different versions of the target software, the version evaluation method of the current version of the target software can be determined by the differential comparison results of each model use case and the corresponding weights.

[0086] Optionally, the weight of each model use case in this embodiment is preset. The weight of the model use case is related to the model category described in the model use case, and model use cases of different model categories have different weights. For example, all model use cases are divided into three categories: customer delivery model, ordinary open source model, and low priority model according to whether they are delivered to customers. The model use cases under the three categories have different weights.

[0087] Furthermore, for multiple model use cases under the same model category, in an optional implementation method, a more detailed classification can be used in this embodiment to assign different weights to different model use cases. For example, a corresponding weight is assigned to the model use case according to the priority level of each model use case. The higher the priority level, the more weight is assigned. Assuming that model use cases under the same category may adopt different scheduling methods, and the priority of the scheduling method is automatic scheduling>manual scheduling>ILP scheduling>quantified priority, correspondingly, the weight of the model use case of the automatic scheduling method is greater than the weight of the model use case of the manual scheduling method, and the same applies to other model use cases. Therefore, the weight of each model use case is determined by a more detailed classification method, so that the weight of each model use case is clearly distinguished, which is conducive to improving the level of detail of the target software evaluation.

[0088] In another optional implementation, each model use case under the same model category in this embodiment may also adopt the same weight. By assigning the same weight to the model use cases of the same model category, while improving the convenience of determining the version evaluation score of the current version of the target software, the evaluation results of the target software can be focused on the model use cases under the model category that have a greater impact on the implementation of the software, which is conducive to optimizing the optimization quality of the target software in a specific usage scenario.

[0089] For ease of explanation, this embodiment uses the same weights for model use cases under the same model category as an example to illustrate the method for determining the target software evaluation score.

[0090] Figure 4 1 is a flow chart of a method for determining a version evaluation score according to an embodiment of the present invention. Figure 4 As shown, in this embodiment, the following steps are included when determining the version evaluation score of the target software.

[0091] In step S410, the model weights corresponding to the model use cases under each model category are determined.

[0092] In this embodiment, the model weight is determined according to the preset scores of each model category and the number of models of the corresponding model use cases, and the value of the model weight is less than 1. Assume that the model use cases corresponding to the target software include three categories: customer delivery model, general open source model and low priority model. The number of model use cases corresponding to the three categories of customer delivery model, general open source model and low priority model are 91, 58 and 22 respectively, and the corresponding preset scores are 70 points, 25 points and 5 points respectively. The model weights under the corresponding categories are 70 / 91 (i.e. 0.76), 25 / 58 (i.e. 0.43) and 5 / 22 (i.e. 0.23), that is, the weight of the model use case of the customer delivery model is 0.76, the weight of the model use case of the general open source model is 0.43, and the weight of the model use case of the low priority model is 0.23.

[0093] Optionally, in this embodiment, the preset score of each model category can be set according to actual usage, with the standard of ensuring that the weight value corresponding to the model category is less than 1.

[0094] In step S420, based on a preset scoring rule, an intermediate score corresponding to each model category is determined according to the differentiated comparison results of the model use cases under each model category and the model weight.

[0095] In this embodiment, after determining the model weights and differentiated comparison results corresponding to the model use cases under each model category, the intermediate scores corresponding to each model category are determined based on the differentiated comparison results and model weights of the model use cases under each model category based on preset scoring rules.

[0096] Optionally, the preset scoring rules in this embodiment include: the initial score of each model use case is determined based on the model weight corresponding to the model category to which it belongs, positive scoring is used for model use cases whose differentiation results are fluctuations not greater than the corresponding reference threshold, negative scoring is used for model use cases whose differentiation results are fluctuations greater than the corresponding reference threshold, and the sum of the initial scores of each model use case under the same category is determined as the middle score of the corresponding model category. For ease of understanding, using the above example, the weight of the model use case of the customer delivery model is 0.76, and the initial score of each model use case is 0.76. Assume that among the 91 customer delivery models, there are 10 model use cases with change rates greater than the reference threshold, 20 model use cases with change rates less than the reference threshold, and 61 model use cases with change rates equal to the reference threshold. That is, the number of model use cases whose differentiated comparison results show that the fluctuation degree is not greater than the corresponding reference threshold is 81 (20+61), and the number of model use cases whose differentiated comparison results show that the fluctuation degree is greater than the corresponding reference threshold is 10. The median score of the customer delivery model is (0.76*81)+(-0.76*10)=53.96.

[0097] Furthermore, the positive scoring part in this embodiment has no upper limit. The negative scoring part is provided with a lower limit. When the negative scoring value exceeds the preset maximum value, the score of the corresponding negative scoring part adopts a fixed value. For example, assuming that the maximum negative score is 20 and the score of the negative scoring part is 30, the calculated score of the negative scoring part adopts -20, and the sum of -20 and the score of the positive scoring part is calculated to determine the middle score of the corresponding model category.

[0098] In step S430, the intermediate scores corresponding to different model categories are summed to determine the version evaluation score of the target software.

[0099] In this embodiment, after determining the intermediate scores corresponding to each model category, the intermediate scores are summed up, and the sum result is determined as the version evaluation score of the corresponding version of the target software.

[0100] Optionally, to facilitate the update iteration of the target software, after determining the version evaluation score of the current version of the target software, the corresponding target software version evaluation method in this embodiment further includes the following steps.

[0101] In step S400, the update status of the current version of the target software is determined according to the version evaluation score and a preset score.

[0102] In this embodiment, by comparing the version evaluation score of the current version of the target software with a preset score, it is determined whether the current version meets the expected requirements, and the update status of the current version of the target software is determined according to the comparison result.

[0103] Optionally, when the version evaluation score of the current version of the target software exceeds the preset score, it indicates that the current version of the target software meets the expected requirements, and the existing target software can be updated through the current version of the target software. Correspondingly, the target software version evaluation method in this embodiment also includes merging the current version of the target software into the code warehouse in response to the version evaluation score of the current version of the target software exceeding the preset score. Among them, the merging operation is an MR (margequest) operation, which initiates a code merging request, modifies the content of this request, adds the content of the current version of the target software to the software stack, adjusts or replaces the original version of the target software in the software stack, and thus achieves the purpose of updating and optimizing the target software.

[0104] Alternatively, when the version evaluation score of the current version of the target software does not exceed the preset score, it indicates that the current version of the target software does not meet the expected requirements, and the current version of the target software can continue to be updated. Correspondingly, the target software version evaluation method in this embodiment also includes updating the current version of the target software in response to the version evaluation score of the current version of the target software not exceeding the preset score. Optionally, when updating the current version of the target software, the function code in the current version of the target software can be modified, or the target software of the existing version in the software stack can be re-modified so that the final target software can meet the expected requirements.

[0105] It should be understood that the initial preset score in this embodiment can adopt a preset empirical value, and after the reference threshold corresponding to each model use case is subsequently determined, it can be adjusted according to the reference threshold of the model use case, or it can be adjusted according to other actual usage situations.

[0106] The technical solution of the embodiment of the present invention compiles and runs each model use case a preset number of times based on different versions of target software to obtain the corresponding numerical distribution range of the target performance indicator, determines the reference threshold value according to the initial numerical distribution range corresponding to the initial version of the target software, and determines the version evaluation score of the current version of the target software according to the differentiated comparison result after comparing the real-time numerical distribution range corresponding to the current version of the target software with the initial numerical distribution range. This can improve the accuracy of the reference threshold value of each model use case, and at the same time, iteratively optimize the target software through the version evaluation score determined above, which is conducive to improving the update quality of the target software version.

[0107] Figure 5 is a flow chart of a method for determining a model use case reference threshold according to an embodiment of the present invention. Figure 5 As shown, in this embodiment, the reference threshold of each model use case is determined through the following steps.

[0108] In step S510, the model use case is compiled and run a preset number of times corresponding to one round based on the initial version of the target software.

[0109] In step S520, an initial value distribution interval corresponding to the target performance indicator cycle in the operation result is counted.

[0110] In step S530, a relative reference value of the model use case is determined according to a preset calculation formula.

[0111] In this embodiment, the preset calculation formula may be the same as the calculation formula for determining the relative reference value described above, which will not be repeated here.

[0112] In step S540, it is determined whether the relative reference value of the model use case is greater than a preset threshold.

[0113] In this embodiment, whether the fluctuation degree of the target performance indicator of the model use case is within the preset allowable fluctuation range is determined by judging whether the relative reference value of the model use case is greater than the preset threshold value. When it is determined that the relative reference value is not greater than the preset threshold value, it indicates that the fluctuation degree of the target performance indicator of the corresponding model use case is within the preset allowable fluctuation range, and step S550 is executed at this time. When it is determined that the relative reference value is greater than the preset threshold value, it indicates that the fluctuation degree of the target performance indicator of the corresponding model use case exceeds the preset allowable fluctuation range, and step S560 is executed at this time.

[0114] In step S550, in response to the relative reference value being not greater than the preset threshold, the preset threshold is determined as a reference threshold for the corresponding model use case.

[0115] In step S560 , in response to the relative reference value being greater than a preset threshold, the relative reference value is determined as a reference threshold corresponding to the model use case.

[0116] In step S570, the reference threshold is written into the configuration file as an attribute of the corresponding model use case.

[0117] The technical solution of this embodiment reflects the target performance index of the corresponding model use case through the numerical distribution interval obtained by multiple compilation and operation of each model use case corresponding to a round, which can eliminate the influence of the fluctuation of the model and hardware environment itself on the target performance index in the model use case operation result to the greatest extent, so that the distribution of the target performance index of each model use case in the current version of the target software is more in line with reality and more valuable, thereby improving the accuracy of determining the reference threshold of each model use case. At the same time, the reference threshold of the corresponding model use case is determined by the relationship between the relative reference value and the preset threshold, and different reference thresholds can be set for each model use case according to the specific usage scenario of the model use case, avoiding the one-size-fits-all situation of using a fixed reference threshold, which can further improve the accuracy of the reference threshold.

[0118] Figure 6 FIG. 1 is a flow chart of a method for updating target software according to an embodiment of the present invention. Figure 6 As shown, in this embodiment, the target software is updated through the following versions.

[0119] In step S610, the new version target software is run.

[0120] In this embodiment, when the update iteration of the target software begins, the new version of the target software is the current version of the target software. The target software corresponds to multiple model use cases. In each operation of a version of the target software, each model use case will be run for a preset number of times corresponding to a round, so as to evaluate and update the target software according to the operation results of the target performance indicators after each model use case runs a round.

[0121] In step S620, the reference threshold of each model use case is read for differential comparison.

[0122] In this embodiment, by reading the reference threshold of each model use case, the real-time mean corresponding to the real-time numerical distribution interval of the target performance indicator obtained under the current version of the target software, and the initial mean corresponding to the initial numerical distribution interval of the target performance indicator obtained under the initial version of the target software, and by comparing the change rate of the real-time mean relative to the initial mean with the reference threshold, the corresponding differentiated comparison result is determined. The specific determination process can refer to the aforementioned related content, which will not be repeated here.

[0123] In step S630, the version evaluation score of the target software is determined by comprehensively considering the use cases of each model.

[0124] In this embodiment, the method for determining the version evaluation score of the current version of the target software may adopt the same method for determining the version evaluation as described above, which will not be described in detail here.

[0125] In step S640, it is determined whether the version evaluation score of the target software reaches a preset score.

[0126] In this embodiment, whether to continue iteratively updating the target software is determined by judging the relationship between the version evaluation score of the current version of the target software and the preset score. When the version evaluation score of the target version is greater than or equal to the preset score, it indicates that the performance of the current version of the target software meets the expected requirements, and the iterative update of the target software can be stopped, and step S650 is continued; when the version evaluation score of the target software is less than the preset score, it indicates that the current version of the target software has not yet met the expected requirements, and it is necessary to continue iteratively updating the target software, and after modifying the function code in the current version to generate a new version of the target software, return to continue to execute step S610, and use the new version of the target software as the current version of the target software in the next iteration process for version evaluation, until the performance of the target software meets the expected requirements, and the overall performance of the target software is optimized and improved.

[0127] In step S650, the target software is merged into the updated version.

[0128] In this embodiment, the old version of the target software is updated and optimized by merging the current version of the target software into the software stack to replace the existing old version of the target software, or by adding new function codes in the current version compared to the old version of the target software to the old version of the target software.

[0129] The technical solution of this embodiment is to run the corresponding model use case through the target software, determine the differentiated comparison results with the corresponding reference thresholds according to the changes in the distribution of the target performance indicators obtained by running each model use case under different versions, determine the version evaluation score of the target software by comprehensively comparing the differentiated comparison results of each model use case, and iteratively optimize the target software according to the version evaluation score, so as to provide an objective basis for the update and iteration of the target software, facilitate the iterative update of the target software, and improve the optimization efficiency and update quality of the target software.

[0130] Figure 7 Schematic diagram of a target software version evaluation device according to an embodiment of the present invention. Figure 7 As shown, the target software in this embodiment corresponds to multiple model use cases, and the target software version evaluation device includes a compilation unit 1, a determination unit 2 and an evaluation unit 3. Among them, the compilation unit 1 is used to compile and run each model use case for a round based on the current version of the target software, and obtain the real-time numerical distribution interval of the target performance indicator of the corresponding model use case, and the round includes a preset number of times. The determination unit 2 is used to compare the comparison result of the real-time numerical distribution interval with the initial numerical distribution interval corresponding to the previous round and the reference threshold of the corresponding model use case, and determine the differentiated comparison result of the corresponding model use case, wherein the reference threshold is determined according to the initial numerical distribution interval corresponding to the initial version of the target software, and is used to characterize the allowable fluctuation degree of the target performance indicator of the corresponding model use case. The evaluation unit 3 is used to determine the version evaluation score of the current version of the target software according to the weight corresponding to each model use case and the differentiated comparison result.

[0131] Figure 8 FIG. 2 is another schematic diagram of the target software version evaluation device according to an embodiment of the present invention. Figure 8 As shown, the determination unit 2 in this embodiment includes a threshold determination module 21 and a difference comparison module 22. Among them, the threshold determination module 21 is used to obtain the initial numerical distribution interval determined after compiling and running each model use case for one round based on the initial version of the target software, determine the relative reference value of the corresponding model use case according to the initial numerical distribution interval, and determine the reference threshold of the corresponding model use case according to the relative reference value. Among them, the relative reference value is used to characterize the degree of fluctuation of the target performance indicator of the corresponding model use case. The difference comparison module 22 is used to determine the real-time mean corresponding to the real-time numerical distribution interval and the initial mean corresponding to the initial numerical distribution interval according to the real-time numerical distribution interval and the initial numerical distribution interval, compare the real-time mean with the initial mean to determine the rate of change of the real-time mean relative to the initial mean, compare the rate of change with the corresponding reference threshold, and determine the differentiated comparison result of the corresponding model use case. Among them, the rate of change is used to characterize the degree of fluctuation of the target performance indicator of the corresponding model use case.

[0132] Optionally, the evaluation unit 3 in this embodiment includes a weight determination module 31, a first score determination module 32 and a second score determination module 33. The weight determination module 31 is used to determine the model weight corresponding to the model use case under each model category. The first score determination module 32 is used to determine the intermediate score corresponding to each model category based on the differentiated comparison results of the model use cases under each model category and the model weight based on the preset scoring rules. The second score determination module 33 is used to sum the intermediate scores corresponding to different model categories to determine the version evaluation score of the target software.

[0133] Furthermore, the target software version evaluation method in this embodiment further includes an incorporation unit 4 and an update unit 5. The incorporation unit 4 is used to, in response to the version evaluation score of the current version of the target software exceeding a preset score, merge the current version of the target software into the code warehouse. The update unit 5 is used to, in response to the version evaluation score of the current version of the target software not exceeding the preset score, perform a version update on the current version of the target software.

[0134] Fig. 9 Schematic diagram of an electronic device according to an embodiment of the present invention. Fig. 9 As shown, Fig. 9 The electronic device shown is a general address query device, which includes a general computer hardware structure, which at least includes a processor 91 and a memory 92. The processor 91 and the memory 92 are connected via a bus 93. The memory 92 is suitable for storing instructions or programs executable by the processor 91. The processor 91 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 91 executes the instructions stored in the memory 92, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 93 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 94 and the display device and the input / output (I / O) device 95. The input / output (I / O) device 95 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 95 is connected to the system via an input / output (I / O) controller 96.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0137] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0138] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0139] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0140] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A target software version evaluation method, characterized in that: The target software corresponds to a plurality of model use cases, and the method comprises: Compile and run each of the model use cases for a round based on the current version of the target software to obtain a real-time numerical distribution interval of the target performance indicator of the corresponding model use case, wherein the round includes a preset number of times; Compare the comparison result of the real-time numerical distribution interval with the initial numerical distribution interval corresponding to the previous round and the reference threshold of the corresponding model use case to determine the differentiated comparison result of the corresponding model use case, wherein the reference threshold is determined according to the initial numerical distribution interval corresponding to the initial version of the target software, and the reference threshold is used to characterize the allowable fluctuation degree of the target performance indicator of the corresponding model use case; The version evaluation score of the current version of the target software is determined according to the weights corresponding to each of the model use cases and the differentiated comparison results.

2. The method according to claim 1, characterized in that The reference threshold is determined based on the following steps: Obtaining an initial value distribution interval determined after compiling and running each of the model use cases for one round based on the initial version of the target software; Determine a relative reference value of the corresponding model use case according to the initial value distribution interval, wherein the relative reference value is used to characterize the degree of fluctuation of the target performance indicator of the corresponding model use case; A reference threshold value corresponding to the model use case is determined according to the relative reference value.

3. The method according to claim 2, characterized in that Determining a reference threshold value corresponding to the model use case according to the relative reference value includes: In response to the relative reference value being not greater than a preset threshold, determining the preset threshold as a reference threshold corresponding to the model use case; In response to the relative reference value being greater than a preset threshold, the relative reference value is determined to be a reference threshold of the corresponding model use case.

4. The method according to claim 1, characterized in that: The comparing the comparison result of the real-time numerical distribution interval with the initial numerical distribution interval corresponding to the previous round and the reference threshold of the corresponding model use case to determine the differentiated comparison result of the corresponding model use case includes: Determine, according to the real-time numerical distribution interval and the initial numerical distribution interval, respectively, a real-time mean corresponding to the real-time numerical distribution interval and an initial mean corresponding to the initial numerical distribution interval; Comparing the real-time mean with the initial mean to determine a rate of change of the real-time mean relative to the initial mean, wherein the rate of change is used to characterize a degree of fluctuation of a target performance indicator of a corresponding model use case; The change rate is compared with the corresponding reference threshold to determine the differentiated comparison result of the corresponding model use case.

5. The method according to claim 1, characterized in that Determining the version evaluation score of the current version of the target software according to the weights and the differentiated comparison results corresponding to the model use cases includes: Determine the model weights corresponding to the model use cases under each model category, wherein the model weights are determined according to the preset scores of each model category and the number of models of the corresponding model use cases; Based on the preset scoring rules, the intermediate scores corresponding to each model category are determined according to the differentiated comparison results of the model use cases under each model category and the model weights; The intermediate scores corresponding to different model categories are summed to determine the version evaluation score of the target software.

6. The method according to claim 5, characterized in that The preset scoring rules include: the initial score of each model use case is determined based on the model weight corresponding to the model category to which it belongs, positive scoring is used for model use cases whose differentiation results have a fluctuation degree no greater than the corresponding reference threshold, negative scoring is used for model use cases whose differentiation results have a fluctuation degree greater than the corresponding reference threshold, and the sum of the initial scores of each model use case under the same category is determined as the middle score of the corresponding model category.

7. The method according to claim 1, characterized in that The method further comprises: In response to the version evaluation score exceeding a preset score, merging the current version of the target software into a code warehouse; or In response to the version evaluation score not exceeding a preset score, the current version of the target software is updated.

8. A target software version evaluation device, characterized in that: The target software corresponds to a plurality of model use cases, and the device comprises: A compiling unit, configured to compile and run a round of each of the model use cases based on the current version of the target software, and obtain a real-time numerical distribution interval of a target performance indicator of the corresponding model use case, wherein the round includes a preset number of times; A determination unit, used to compare the comparison result of the real-time numerical distribution interval with the initial numerical distribution interval corresponding to the previous round and the reference threshold of the corresponding model use case, and determine the differentiated comparison result of the corresponding model use case, wherein the reference threshold is determined according to the initial numerical distribution interval corresponding to the initial version of the target software, and the reference threshold is used to characterize the allowable fluctuation degree of the target performance indicator of the corresponding model use case; An evaluation unit is used to determine a version evaluation score of the current version of the target software according to the weights corresponding to each of the model use cases and the differentiated comparison results.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 7 are implemented.