Performance Evaluation System

Abstract

To provide a performance evaluation system capable of further improving efficiency of performance evaluation of an evaluation target.SOLUTION: A performance evaluation system 10 includes: a test unit 12 tests an evaluation target according to the indicated test conditions; an evaluation unit 16 that evaluates performance of the evaluation target on the basis of the result of the test and outputs individual evaluation results; and a condition setting model 18 that performs machine learning using the test conditions and the individual evaluation results from past tests as teacher data and outputs test conditions for a next test. The output of the test condition by the condition setting model 18, the test by the outputted test condition, and the evaluation of the result of the test are repeated until a predetermined end condition is met.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This specification discloses a performance evaluation system for evaluating the performance of an object to be evaluated and an information processing apparatus used in the performance evaluation system.

Background Art

[0002] Generally, when developing a product, it is necessary to repeat the operations of testing a prototype, identifying development issues from the evaluation results of the test, and correcting the development issues. For example, when developing a vehicle, a test driver performs a test drive of the vehicle, evaluates the performance of the vehicle based on the values of various evaluation indices obtained by the test drive, extracts the indices with poor evaluation results as development issues, and repeats the operation of improving them.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such tests of an object to be evaluated, test conditions (such as a speed profile, etc.) are set so that more development issues can be extracted in one test. Conventionally, such test conditions have often been set intuitively by a test performer (such as a test driver, etc.) based on their experience. Therefore, depending on the experience of the test performer, appropriate test conditions may not be set, and as a result, performance evaluation may not be efficiently performed in some cases.

[0005] Note that Patent Document 1 discloses a technique for virtually testing a test object for efficient product development and automatically controlling and adjusting the actual test object according to the results. According to such a technique, the burden of product development can be reduced to a certain extent. However, in the technique of Patent Document 1, how to set the test conditions has not been studied. Therefore, with the technique of Patent Document 1, it is not possible to test under appropriate conditions, and as a result, the number of repetitions of the test operation may increase, or important development issues may not be extracted. And as a result, with the technique of Patent Document 1, the efficiency of performance evaluation may decrease.

[0006] Therefore, this specification discloses a performance evaluation system that can further improve the efficiency of performance evaluation of an evaluation object, and an information processing apparatus used in the performance evaluation system.

Means for Solving the Problems

[0007] The performance evaluation system disclosed in this specification includes a test apparatus that tests an evaluation object according to specified test conditions, an evaluation apparatus that evaluates the performance of the evaluation object based on the results of the test and outputs individual evaluation results, and a condition setting model that performs machine learning on the test conditions and the individual evaluation results in past tests as teacher data and outputs the test conditions for the next test. Until a predetermined end condition is satisfied, the output of the test conditions by the condition setting model, the test according to the output test conditions, and the evaluation of the results of the test are repeated.

[0008] With such a configuration, appropriate test conditions can be set without depending on the operator's experience. And thereby, the efficiency of performance evaluation of the evaluation object can be further improved.

[0009] In this case, prior to the output of the test conditions from the condition setting model, the test apparatus executes an initial test which is a test according to a plurality of initial test conditions set by the fractional factorial design method, the evaluation apparatus outputs an initial individual evaluation result which is the individual evaluation result of the initial test, and the condition setting model may perform initial learning which is machine learning using the plurality of initial test conditions and the plurality of initial individual evaluation results as teacher data prior to the output of the test conditions.

[0010] With such a configuration, data necessary for initial learning can be collected by the performance evaluation system.

[0011] In this case, after completion of the initial learning, the condition setting model may continue machine learning using the test conditions output by the condition setting model and the individual evaluation result corresponding to the test conditions as teacher data.

[0012] With such a configuration, the quality of search for test conditions can be further improved.

[0013] Further, the test apparatus has a PT-VRS in which a power train of a vehicle and a vehicle model that virtually realizes vehicle dynamic characteristics are combined, and the test conditions may include a profile of vehicle speed change and gradient data of a road surface.

[0014] With such a configuration, various performances related to the vehicle can be evaluated with a simple configuration.

[0015] Further, the condition setting model may output, as future test conditions, conditions predicted to obtain low individual evaluation results.

[0016] With such a configuration, development issues can be efficiently found.

[0017] The information processing apparatus disclosed in this specification is configured to function as an evaluation apparatus that evaluates the performance of an object to be evaluated based on the result of testing the object to be evaluated according to specified test conditions and outputs an individual evaluation result, and a condition setting model that performs machine learning using the test conditions and the individual evaluation results in past tests as teacher data and outputs the test conditions for the next test.

[0018] With such a configuration, appropriate test conditions can be set without depending on the experience of the operator. As a result, the efficiency of the performance evaluation of the object to be evaluated can be further improved.

Effect of the Invention

[0019] According to the technology disclosed in this specification, the efficiency of the performance evaluation of the object to be evaluated can be further improved.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Best Mode for Carrying Out the Invention

[0021] Hereinafter, the configuration of the performance evaluation system 10 will be described with reference to the drawings. FIG. 1 is a block diagram showing the basic configuration of the performance evaluation system 10. This performance evaluation system 10 is a system for evaluating the performance of an object to be evaluated (for example, a product such as a vehicle).

[0022] The performance evaluation system 10 includes a test device 12 and an information processing device 14. Further, the information processing device 14 functions as an evaluation device 16 and a condition setting model 18.

[0023] The test device 12 tests the object to be evaluated according to the specified test conditions, obtains the values of a plurality of parameters (hereinafter referred to as "target parameters") regarding the behavior or properties of the object to be evaluated, and outputs this as a test result. Here, the test of the object to be evaluated may be performed using the actual object to be evaluated, or a part or all of the object to be evaluated may be replaced with a virtual model (for example, a mathematical model, etc.) and the test may be performed. In the former case, the test device 12 includes a measuring instrument for measuring the target parameters. In the latter case, part or all of the test of the object to be evaluated becomes a simulation that is virtually performed on a computer. In this case, the values of some or all of the plurality of target parameters are calculated during the simulation process.

[0024] Physically, the information processing device 14 is a computer having a processor and a memory. This "computer" also includes a microcontroller in which a computer system is incorporated into one integrated circuit. Further, the information processing device 14 may be configured by combining a plurality of computers instead of a single computer. Therefore, the evaluation device 16 and the condition setting model 18 may be different computers from each other. Also, when testing, when part or all of the object to be evaluated is replaced with a virtual model, this information processing device 14 may function as part or all of the test device 12.

[0025] The information processing device 14 functionally functions as an evaluation device 16 and a condition setting model 18. The evaluation device 16 evaluates the performance of the object to be evaluated based on the test results (i.e., the values of a plurality of target parameters). Here, the performance evaluation performed by the evaluation device 16 is an evaluation based on a single test result. On the other hand, the performance evaluation system 10 evaluates the performance of the object to be evaluated from the results of performing tests multiple times. Therefore, hereinafter, the performance evaluation by the evaluation device 16 is referred to as "individual evaluation", the performance evaluation by the performance evaluation system 10 is referred to as "comprehensive evaluation", and the two are distinguished.

[0026] For individual evaluation, the evaluation device 16 stores evaluation functions corresponding to each of a plurality of evaluation indicators. The evaluation function is a function having the target parameters obtained in the test as variables and is a function that quantitatively represents the corresponding evaluation indicator. For example, as evaluation indicators for vehicle drivability (hereinafter referred to as "drivability"), there are linearity during acceleration and shock feeling during gear shifting. Among these, with regard to the linearity during acceleration, the higher the linearity of the longitudinal acceleration of the vehicle with respect to the accelerator pedal operation amount, the better. Therefore, when the evaluation device 16 evaluates the drivability of a vehicle, the evaluation device 16 stores, as the "evaluation function for linearity during acceleration", a function that obtains a higher score as the linearity of the longitudinal acceleration of the vehicle with respect to the accelerator pedal operation amount is higher. Such a plurality of evaluation functions are set in advance by an operator.

[0027] When the evaluation device 16 obtains a test result from the test device 12, the evaluation device 16 applies this test result to the evaluation function and calculates scores for a plurality of evaluation items. Then, the evaluation device 16 transmits the calculated plurality of scores to the condition setting model 18 as individual evaluation results.

[0028] The condition setting model 18 is an artificial intelligence that performs machine learning using past test conditions and individual evaluation results as teacher data, determines future test conditions, and outputs them. Here, the test conditions are the conditions when testing the object to be evaluated. For example, when testing the drivability of a vehicle, the vehicle speed change profile and road surface gradient data of the vehicle are test conditions.

[0029] The condition setting model 18 is programmed to preferentially output test conditions that can more efficiently perform a comprehensive evaluation of the performance of the object to be evaluated. For example, consider the case of performing a comprehensive evaluation of a vehicle for the purpose of finding development issues of the vehicle. A development issue refers to matters that need to be improved in the vehicle, that is, matters with low performance and thus low evaluation scores. For efficient development, it is required to find more development issues with fewer test runs. Therefore, in such a case, the condition setting model 18 searches for test conditions under which more development issues can be found, in other words, test conditions under which the evaluation score is expected to be low. The test conditions output by the condition setting model 18 are sent to the test apparatus 12. When the test apparatus 12 receives the test conditions from the condition setting model 18, it tests the object to be evaluated according to the test conditions.

[0030] The condition setting model 18 has previously machine-learned past test conditions and individual evaluation results as teacher data. The teacher data used for this learning may be provided by an operator or may be acquired using the test apparatus 12 and the evaluation apparatus 16. For example, prior to the output of test conditions by the condition setting model 18, the test apparatus 12 may perform tests and the evaluation apparatus 16 may perform individual evaluations multiple times, and the test conditions and individual evaluation results at that time may be used as teacher data. The test conditions used in the tests for acquiring teacher data may be provided by an operator or may be automatically calculated in the test apparatus 12 or the information processing apparatus 14 according to the experimental design method.

[0031] The form of machine learning performed by the condition setting model 18 is not particularly limited. However, the condition setting model 18 in this example performs supervised learning. Therefore, the condition setting model 18 may perform machine learning using linear regression, logistic regression, random forest, boosting, support vector machine, neural network, autoregressive model, or a combination thereof. In this example, the condition setting model 18 is an artificial intelligence constructed according to the theory of Bayesian Active Learning (hereinafter referred to as "BAL").

[0032] The algorithm of this BAL will be described. In BAL, the design variable Xi suitable for obtaining the desired response Yi is explored. In this example, the test condition is the "design variable xi", and the score of a specific evaluation index corresponds to the "response yi". The algorithm for exploring the optimal design variable Xi is as follows.

[0033]

Equation

[0034] Next, with reference to FIG. 2, the process of comprehensively evaluating the performance of the object to be evaluated using this performance evaluation system 10 will be described. When comprehensively evaluating the performance of the object to be evaluated, first, the condition setting model 18 is made to perform initial learning (S10). Specifically, the test conditions (hereinafter referred to as "initial test conditions") and individual evaluation results (hereinafter referred to as "initial individual evaluation results") prepared in advance for learning are input into the condition setting model 18 as teacher data for machine learning. Note that these initial test conditions and initial individual evaluation results may actually be obtained by testing the object to be evaluated, or may be input by the operator.

[0035] Once the initial learning is completed, the condition setting model 18 determines the test conditions that are predicted to obtain the desired individual evaluation results as the next test conditions and transmits them to the test apparatus 12 (S12). At this time, the condition setting model 18 searches for test conditions that can efficiently evaluate the performance based on the teacher data. For example, when performing a comprehensive evaluation for the purpose of discovering development issues, it searches for test conditions where low individual evaluations are expected. As described above, usually, there are multiple evaluation indicators. Therefore, the condition setting model 18 may change the evaluation indicators to be emphasized at any time when calculating the test conditions. For example, in the first n times, it searches for and outputs test conditions where the first evaluation indicator is low, and in the next n times, it searches for and outputs test conditions where the second evaluation indicator is low. In this way, in the search for test conditions, the target evaluation index is hereinafter referred to as "KPI".

[0036] When the test apparatus 12 receives the test conditions from the condition setting model 18, it executes the test of the object to be evaluated according to the test conditions (S14). Then, the value of the target parameter obtained in this test is transmitted to the evaluation apparatus 16 as the test result. The evaluation apparatus 16 individually evaluates the object to be evaluated based on the received test result (S16). Once the individual evaluation is completed, the condition setting model 18 determines whether a predetermined end condition is satisfied (S18). The end condition is a condition for ending the comprehensive evaluation process of the object to be evaluated. For example, it is the number of times of test execution when determining to end the comprehensive evaluation process. When the end condition is satisfied, the comprehensive evaluation process of the object to be evaluated ends.

[0037] On the other hand, when the end condition is not satisfied, the condition setting model 18 performs additional machine learning using the test conditions and their individual evaluation results used in the previous test as teacher data (S20). Then, it returns to step S12, and the condition setting model 18 outputs the next test conditions. Thereafter, until the end condition is satisfied, the output of the test conditions, the test execution, the individual evaluation, and the additional learning are repeated.

[0038] As is clear from the above description, in this example, the following test conditions are determined using the machine-learned condition setting model 18 (i.e., artificial intelligence). As a result, appropriate test conditions can be explored without relying on the operator's experience, and the comprehensive evaluation of the performance of the object to be evaluated can be efficiently performed. Further, in this example, the test device 12, the evaluation device 16, and the condition setting model 18 are configured to communicate data with each other. Therefore, the test of the object to be evaluated, the individual evaluation, and the setting of the test conditions can be automatically repeated, reducing the burden on the operator and the time required for the comprehensive evaluation.

[0039] Next, a specific example of this performance evaluation system 10 will be described. FIG. 3 is a block diagram showing the configuration of a performance evaluation system 10 for evaluating the performance of a vehicle. This performance evaluation system 10 also includes a test device 12 and an information processing device 14. The test device 12 is a power train virtual and real simulator (hereinafter referred to as "PT-VRS22") that combines a power train 23 and a model that virtually realizes vehicle dynamic characteristics. PT-VRS22 has an actual machine of the power train 23. It also has a vehicle controller 40 that controls the drive of the power train 23. The vehicle controller 40 has substantially the same configuration as the vehicle controller 40 (so-called ECU) mounted on an actual vehicle.

[0040] The power train 23 includes a prime mover 24, a transaxle 26, and a drive shaft 28. Dynamometers 30 and 34 that replace the tires and brakes are attached to both ends of the drive shaft 28.

[0041] The vehicle controller 40 controls the drive of the power train 23 based on a command signal Sg1 sent from the control unit 36. Further, the vehicle controller 40 acquires detection values (e.g., engine speed, etc.) from various sensors mounted on the power train 23 as a vehicle control value Sg3 and outputs it to the evaluation device 16.

[0042] The driver model 37 is a mathematical model that simulates a driver. Based on the vehicle speed profile SP described later and the vehicle speed Sg5 input from the control unit 36, the driver model 37 calculates the accelerator / brake amount Sg4 indicated by the driver and outputs it to the control unit 36. The vehicle model 38 is a mathematical model that simulates a vehicle. Based on the gradient data RD described later and the torque on the drive shaft 28 (hereinafter referred to as "D / S torque") Sg7 input from the control unit 36, the vehicle model 38 calculates the wheel speed Sg6 of the vehicle and outputs it to the control unit 36.

[0043] Based on the accelerator / brake amount Sg4, the control unit 36 generates and outputs a command signal Sg1 to the vehicle controller 40 and a command signal Sg8 to the dynamometers 30, 34. Also, the control unit 36 calculates the D / S torque Sg7 from the measured values Sg9 at the dynamometers 30, 34 and outputs it to the vehicle model 38. Furthermore, the control unit 36 calculates the vehicle speed Sg5 based on the wheel speed Sg6 obtained from the vehicle model 38 and outputs it to the driver model 37. Furthermore, the control unit 36 outputs the vehicle speed Sg5 and the acceleration Sg10 obtained from the vehicle speed Sg5 to the evaluation device 16.

[0044] The evaluation device 16 individually evaluates the drivability of the vehicle based on the target parameters output from the test device 12, specifically, the values of the vehicle speed Sg5, the acceleration Sg10, and the vehicle control value Sg3. Specifically, the evaluation device 16 stores evaluation functions corresponding to a plurality of evaluation indices related to drivability (for example, the linearity during acceleration, the shock feeling during gear shifting, etc.). The evaluation device 16 applies the values of the target parameters output from the test device 12 to this evaluation function and calculates an evaluation score for each evaluation index. Then, the evaluation device 16 outputs the calculated plurality of evaluation scores to the condition setting model 18 as individual evaluation results.

[0045] The condition setting model 18 determines test conditions and outputs them to the test apparatus 12. Before the completion of the initial learning, the condition setting model 18 determines and outputs initial test conditions according to the design of experiments. The test apparatus 12 performs a vehicle test according to the initial test conditions, and the evaluation apparatus 16 individually evaluates the results of the test and outputs the results as initial individual evaluation results. The condition setting model 18 performs machine learning, that is, initial learning, using the initial test conditions and the initial individual evaluation results as teacher data. When the initial learning is completed, the condition setting model 18 determines the next test conditions according to the BAL algorithm and outputs them to the test apparatus 12. In addition, each time a test is completed, the condition setting model 18 performs additional machine learning using the test conditions and the individual evaluation results as teacher data.

[0046] In this example, the test conditions include a vehicle speed profile SP and gradient data RD. FIG. 4 is a diagram showing an example of the vehicle speed profile SP. In this example, the vehicle speed profile SP includes a first speed V1 which is the initial vehicle speed, a second speed V2 which is the next vehicle speed, a third speed V3 which is the next vehicle speed after that, a first time ΔT1 which is the time from the first speed V1 to the change to the second speed V2, a second time ΔT2 which is the time of constant speed running at the second speed V2, and a third time ΔT3 which is the time from the second speed V2 to the change to the third speed V3. The gradient data RD is data representing the gradient of the road surface on which the vehicle travels. Further, the test conditions may include control parameters EP of the vehicle controller 40. The control parameter EP is a control parameter when the vehicle controller 40 drives and controls the power train 23, for example, a control gain or a constant of a mathematical formula used for calculating a command value.

[0047] The information processing apparatus 14 further includes a U / I device 42 and a communication I / F 44. The U / I device 42 includes an input device that receives operations from an operator and an output device that presents information to the operator. The operator sets preconditions via this U / I device 42. The preconditions are basic conditions for comprehensively evaluating the performance of the object to be evaluated. Such preconditions include, for example, the numerical range of settable test conditions, the number of test executions, the types of evaluation metrics used for individual evaluations, and the like.

[0048] The communication I / F 44 is an interface for data communication with the test apparatus 12. This data communication may be wired or wireless. The communication I / F 44 transmits the vehicle speed profile SP, gradient data RD, and control parameter EP output from the condition setting model 18 to the test apparatus 12 as test conditions.

[0049] Next, the test conditions calculated by the condition setting model 18 will be described. FIG. 5 is a diagram showing an example of initial test conditions and normal test conditions output after completion of initial learning. In FIG. 5, the black circles indicate the initial test conditions, and the crosses indicate the normal test conditions. Also in FIG. 5, the horizontal axis represents the first characteristic A defined by the test conditions, and the vertical axis represents the second characteristic B defined by the test conditions. Furthermore, the density of the ink hatching in FIG. 5 indicates the individual evaluation results for a specific evaluation metric, and the darker the ink hatching, the lower the individual evaluation result.

[0050] As shown in FIG. 5, the condition setting model 18 determines a plurality of initial test conditions according to the experimental design method so that the first characteristic A and the second characteristic B vary evenly. Since the initial test conditions are determined regardless of the individual evaluation results, they also include many conditions that can obtain high evaluation results, for example, test conditions such as points P1 and P2.

[0051] The condition setting model 18 performs machine learning using this initial test condition and the individual evaluation results of the initial test as teacher data, and calculates test conditions that can efficiently comprehensively evaluate the object to be evaluated. As a result, after the completion of the initial learning, the condition setting model 18 preferentially searches around points P3, P4, and P5 where low individual evaluation results are likely to be obtained, and preferentially outputs the test conditions around points P3, P4, and P5. And thereby, tests under conditions where the individual evaluation becomes low are preferentially executed, and the development issues of the object to be evaluated can be efficiently discovered.

[0052] Next, the flow of the comprehensive evaluation of the vehicle performance by the performance evaluation system 10 in FIG. 3 will be described with reference to FIGS. 6 and 7. In this case, the operator first sets the preconditions via the U / I device 42 (S30). The preconditions include, as described above, the numerical range of the test conditions that can be set, the number of test executions, the types of evaluation indicators used for individual evaluation, the allowable range of the initial state quantity of the object to be evaluated, and the like. If the preconditions can be set, the condition setting model 18 determines the initial test conditions according to the experimental design method and outputs them to the test device 12 (S32). The test device 12 checks whether the initial state quantity of the vehicle (accurately, the power train 23 and the PT-VRS 22 that models vehicle elements other than the power train 23) (for example, the water temperature of the engine, etc.) is within the allowable range (S34). If the initial state quantity exceeds the appropriate range, the object to be evaluated and the test equipment are adjusted so as to be within the appropriate range (S36).

[0053] When the initial state quantity is appropriate, the test device 12 executes an initial test (S38). Specifically, the vehicle speed profile SP, gradient data RD, and control parameter EP that constitute the initial test conditions are input to and operated on each part of the test device 12. During this initial test period, the test device 12 outputs the measured values of predetermined target parameters (such as vehicle speed, etc.) as test results to the evaluation device 16. Note that depending on the content of the initial test conditions, there may be a case where an error occurs without the power train 23 operating even once. Also in this case, so that the values of target parameters such as vehicle speed can be output as test results, the test device 12 starts saving the values of the target parameters before the operation start of the power train 23.

[0054] When the initial test is completed, the evaluation device 16 individually evaluates the obtained initial test results and outputs the results as initial individual evaluation results (S40). The condition setting model 18 performs machine learning using the initial test conditions and the initial individual evaluation results as teacher data (S42). Subsequently, the condition setting model 18 determines whether the initial learning has been completed (S44). Specifically, the condition setting model 18 determines whether the machine learning using the teacher data for a predefined number of targets has been completed. As a result of the determination, if the initial learning is not completed, the condition setting model 18 returns to step S32 and outputs the initial test conditions again based on the planned experiment method. Thereafter, steps S32 to S44 are repeated until the initial learning is completed.

[0055] When the initial learning is completed, as shown in FIG. 7, the condition setting model 18 determines and outputs the next test conditions by BAL (S46). If the test device 12 receives the next test conditions, it determines the suitability of the initial state quantity (S48), adjusts the initial state quantity (S50), and executes a test according to the test conditions (S52), respectively. When the test is completed, the evaluation device 16 performs an individual evaluation of the vehicle based on the transmitted test results (S54). Specifically, the evaluation device 16 applies the test results to a plurality of evaluation functions and calculates evaluation scores for each of the plurality of evaluation indicators.

[0056] Once the evaluation score is calculated, the condition setting model 18 checks whether the termination condition is satisfied (S56). As a result of the check, if the termination condition is satisfied, the overall evaluation process of the vehicle's performance ends. On the other hand, if the termination condition is not satisfied, the condition setting model 18 performs additional learning using the test conditions and individual evaluation results used in the previous test as teacher data (S58). After the additional learning, the condition setting model 18 determines the next test conditions according to the BAL algorithm and transmits them to the test device 12 (S60). Thereafter, the processes of steps S48 to S60 are repeated until the termination condition is satisfied. In BAL, a predetermined evaluation index, that is, a test condition under which the score of the KPI becomes low is searched for and output. This KPI is changed periodically or aperiodically. For example, in the first 10 times, test conditions under which the evaluation index regarding the linearity during acceleration becomes low may be searched for, and in the next 10 times, test conditions under which the evaluation index regarding the shock feeling during gear shifting becomes low may be searched for.

[0057] In this way, in this example, test conditions that enable efficient overall evaluation are searched for by using the condition setting model 18 (that is, artificial intelligence) that has machine-learned the results of past tests. As a result, development issues can be discovered efficiently, and the costs and time required for the overall evaluation of the vehicle can be reduced. Further, since the test conditions are searched for by artificial intelligence, the performance can be evaluated under appropriate conditions even if the person (operator) performing the test lacks experience. As a result, variations in the quality of the overall evaluation due to differences in the experience of the operator can be suppressed. Also, in this example, the test device 12, the evaluation device 16, and the condition setting model 18 can communicate data with each other. Therefore, the determination of the test conditions, the test of the vehicle, and the individual evaluation of the test results can be automatically and continuously repeated. As a result, the labor and time required for the overall evaluation of the vehicle's performance can be reduced.

[0058] Next, other application examples of the performance evaluation system 10 will be described. When the vehicle is traveling at a steady speed or accelerating gently, the floor of the vehicle may vibrate at a specific frequency. In order to suppress such vibration of the floor, vibration control for reducing torsional vibration of the drive system is known, which uses a driving motor provided in the power train of a hybrid electric vehicle. Hereinafter, such a vibration control system will be referred to as a "motor utilization vibration control system". In the motor utilization vibration control system, the driving motor is feedback-controlled to reduce the torsional vibration of the drive system.

[0059] The performance evaluation system 10 can also be used for the comprehensive evaluation of this motor utilization vibration control system. In this case, the configuration of the performance evaluation system 10 is almost the same as the configuration shown in FIG. 3. However, the power train 23 has an engine and a driving motor as power sources, and the vehicle controller 40 feedback-controls the driving motor to reduce the torsional vibration of the drive system. In addition, the test conditions include two time constants Kx and Ky, in addition to the vehicle speed profile SP, the gradient data RD, and the control parameter EP. The time constants Kx and Ky are the time constants used in the feedback control of the driving motor. The performance evaluation system 10 comprehensively evaluates the performance of the motor utilization vibration control system in the same flow as the flowcharts shown in FIGS. 6 and 7.

[0060] As a result of performing such a comprehensive evaluation, it becomes possible to specify the time constants Kx and Ky suitable for vibration suppression. FIG. 8 is a diagram for explaining the specification of the time constants Kx and Ky using the result of the comprehensive evaluation in the performance evaluation system 10.

[0061] In FIG. 8, graph Ga shows the results when the test conditions are explored with the first evaluation index E1 as the KPI. The first evaluation index E1 is one of the indexes representing the performance of the motor utilization vibration control system, and it is an index that is required to fall within a predetermined allowable range. The condition setting model 18 explores the test conditions such that the first evaluation index E1 becomes as low as possible within the allowable range. The circles in graph Ga are the test conditions output from the condition setting model 18 as a result of exploring with the first evaluation index E1 as the KPI. Among these, the black circles are the test conditions where the first evaluation index E1 is within the allowable range and near the lower limit value of the allowable range. This black circle is the boundary point of the allowable time constants Kx and Ky, and the curve L1 approximating the black circle becomes the boundary line of the allowable time constants Kx and Ky. In graph Ga, the hatched area is the area where the first evaluation index E1 is within the allowable range, and the white area is the area where the first evaluation index E1 is outside the allowable range.

[0062] In order to efficiently specify appropriate time constants Kx and Ky, it is required that such a boundary line L1 can be accurately estimated with a small number of test times. As is clear from referring to graph Ga, in this example, most of the test conditions (circles) output from the condition setting model 18 are the boundary points (black circles) of the time constants Kx and Ky. From this, it can be seen that according to this example, the boundary line L1 of the first evaluation index E1 can be accurately estimated with a small number of test times.

[0063] Graph Gb in FIG. 8 shows the results when the test conditions are explored with the second evaluation index E2 as the KPI. As is clear from referring to graph Gb, even when the second evaluation index E2 is used as the KPI, most of the test conditions (circles) output from the condition setting model 18 are the boundary points (black circles) of the time constants Kx and Ky. From this, it can be seen that even when the second evaluation index E2 is used as the KPI, its boundary line L2 can be accurately estimated with a small number of test times.

[0064] The graph Gc in FIG. 8 is a graph showing the achievable range in accordance with the specifications of the components of the motor-utilizing vibration control system. In the graph Gc, the hatched area indicates the achievable area, and the white area indicates the unachievable area. Such an achievable area can be specified from the specifications of the components of the motor-utilizing vibration control system.

[0065] The graph Gd in FIG. 8 is a graph showing the AND area of the hatched areas in the three graphs Ga, Gb, and Gc. For the hatched area in the graph Gd, both the first evaluation index E1 and the second evaluation index E2 are within the allowable range, and the specifications of the components of the motor-utilizing vibration control system can be satisfied. Therefore, it can be understood that when it is desired to specify the time constants Kx and Ky in the development process of the motor-utilizing vibration control system, points within the hatched area in the graph Gd may be selected. Thus, by using the performance evaluation system 10 of this example, the selectable range of the time constants Kx and Ky can be easily specified, and even a developer with little experience can easily and appropriately set the time constants Kx and Ky.

[0066] Also, the performance evaluation system 10 of this example can also be used for evaluating the noise vibration performance (hereinafter referred to as "NV" performance) of a vehicle. Regarding this, an example will be described using the "howl" generated by the CVT60, which is one of the components of the transaxle 26. FIG. 9 is a schematic diagram of the CVT60. The howl is a phenomenon in which the string vibration of the steel belt 64 used in the CVT60 becomes the excitation source, and a strange noise of "howl" occurs during hard acceleration.

[0067] Such a howl is caused, in part, by the fact that as the winding diameter D of the steel belt 64 varies, when the steel belt 64 moves in the radial direction of the sheave 62, it does not move smoothly, and intermittent motion in which slipping and sticking occur alternately, so-called stick-slip, occurs. When stick-slip occurs, the string portion of the steel belt 64 vibrates, generating a howl.

[0068] Therefore, when comprehensively evaluating the performance related to the horn sound of a vehicle, a measuring instrument for measuring the horn sound is provided in the test apparatus 12, and the measured value is output to the evaluation apparatus 16 as one of the test results. Also, the test conditions include parameters related to the behavior of the CVT60, such as the vehicle speed profile SP, etc. After the completion of the initial learning, the condition setting model 18 searches for and outputs test conditions under which the horn sound is likely to occur according to the BAL algorithm. In this case, the condition setting model 18 calculates the test conditions using evaluation indexes related to sound and vibration as KPIs. Thereby, the developer can grasp under what conditions the horn sound is likely to occur with a small number of test times, and can efficiently consider countermeasures for suppressing the horn sound.

[0069] Also, all the configurations described so far are just examples. If the next test conditions are determined by a condition setting model that has been machine-learned in advance, other configurations may be changed as appropriate. Therefore, the performance evaluation system 10 of this example may be used not only for vehicles but also for comprehensive evaluation of the performance of other products, such as electrical appliances, electronic components, materials, etc.

Explanation of Signs

[0070] 10 Performance evaluation system, 12 Test apparatus, 14 Information processing apparatus, 16 Evaluation apparatus, 18 Condition setting model, 22 PT-VRS, 23 Power train, 24 Prime mover, 26 Transaxle, 28 Drive shaft, 30, 34 Dynamometer, 36 Control unit, 37 Driver model, 38 Vehicle model, 40 Vehicle controller, 42 U / I device, 44 Communication I / F, 60 CVT, 62 Sheave, 64 Steel belt, RD Gradient data, SP Vehicle speed profile.

Claims

1. A test apparatus for testing a vehicle as an object to be evaluated according to specified test conditions, the test apparatus having a PT-VRS that combines a power train of the vehicle and a vehicle model that virtually realizes vehicle dynamic characteristics, an evaluation apparatus that evaluates the performance of the object to be evaluated based on the results of the test and outputs a plurality of individual evaluation results, a condition setting model that performs machine learning using the test conditions and the individual evaluation results in past tests as teacher data and outputs test conditions for the next test, and the test condition output by the condition setting model, the test according to the output test conditions, and the evaluation of the test results are repeated until a predetermined end condition is satisfied, wherein the test conditions include a profile of vehicle speed change and gradient data of the road surface, and the condition setting model outputs, as future test conditions, conditions predicted to result in a lower value of an evaluation index, which is one of the plurality of individual evaluation results. A performance evaluation system characterized by the above.

2. The performance evaluation system according to claim 1, wherein the test apparatus executes an initial test, which is a test according to a plurality of initial test conditions set by a planned experiment method, prior to the output of the test conditions from the condition setting model, the evaluation apparatus outputs initial individual evaluation results, which are the individual evaluation results of the initial test, and the condition setting model performs initial learning, which is machine learning using the plurality of initial test conditions and the plurality of initial individual evaluation results as teacher data, prior to the output of the test conditions. A performance evaluation system characterized by the above.

3. The performance evaluation system according to claim 2, wherein after completion of the initial learning, the condition setting model continues machine learning using the test conditions output by the condition setting model and the individual evaluation results corresponding to the test conditions as teacher data. A performance evaluation system characterized by the above.

4. The performance evaluation system according to claim 1, wherein the condition setting model periodically or aperiodically changes the evaluation index. A performance evaluation system characterized by the above.

5. The performance evaluation system according to claim 1, wherein the test apparatus, the evaluation apparatus, and the condition setting model can communicate data with each other, and the test apparatus tests the object to be evaluated according to the test conditions transmitted from the condition setting model. The evaluation device outputs a plurality of the individual evaluation results based on the target parameters transmitted from the test device. The condition setting model outputs the test conditions for the next test based on the plurality of the individual evaluation results transmitted from the evaluation device and the test conditions transmitted to the test device. A performance evaluation system characterized by the above.

Images