Inspection Control Device

The inspection control device uses reinforcement learning to optimize the powertrain's operating state for efficient abnormality detection, addressing the limitations of manual condition selection and improving inspection efficiency.

JP7593343B2Active Publication Date: 2024-12-03TOYOTA JIDOSHA KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022008716
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-01-24
Publication Date
2024-12-03
Estimated Expiration
2042-01-24

AI Technical Summary

Technical Problem

Existing inspection methods for vehicle powertrains are limited by the need to manually select measurement conditions from predetermined patterns, which may not be optimal and require significant time and resources, and fail to efficiently shift the powertrain to an operating state suitable for inspection.

Method used

An inspection control device using reinforcement learning to optimize the powertrain's operating state by controlling it with a learned model that maximizes the detection of abnormal sounds or vibrations, reducing the number of actions required to achieve an optimal inspection condition.

Benefits of technology

The device quickly shifts the powertrain to an operating state suitable for inspection, enhancing detection accuracy and efficiency by minimizing manual intervention and optimizing the selection of measurement conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593343000004
    Figure 0007593343000004
  • Figure 0007593343000005
    Figure 0007593343000005
  • Figure 0007593343000006
    Figure 0007593343000006
Patent Text Reader

Abstract

To provide an inspection controller which can rapidly operate a power train to inspect into a driving state suitable for an inspection.SOLUTION: An inspection controller 100 includes: a storage device 130 for storing a learned model which selects an action of operating the amount of controlling a power train and outputting the action when a state variable showing the operation state of the power train is input; and a processing circuit 110 for operating a control amount on the basis of the output action by inputting the state variable into the learned model and controlling the power train. The learned model is a model learned by a reinforcing learning which gives a reward so that a benefit becomes larger as the power train is shifted with less actions into an operating state in which sound sent from the power train is suitable for determining the presence or absence of abnormalities.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an inspection control device that is applied to an inspection system for inspecting a powertrain of a vehicle and controls the powertrain. [Background technology]

[0002] There is a known inspection method for detecting the occurrence of abnormal sounds or vibrations based on data detected by a sensor. It is preferable to perform such an inspection under measurement conditions that make it easy to detect abnormal sounds or vibrations.

[0003] Patent Document 1 discloses an assistance device that makes it easy to identify measurement conditions that are easy to detect abnormal sounds and abnormal vibrations. The assistance device disclosed in Patent Document 1 creates heat maps of feature amounts for data under various measurement conditions based on data measured under each measurement condition. Then, the multiple heat maps are arranged in a matrix on a two-axis orthogonal coordinate system and displayed, making it easy to identify measurement conditions that are easy to detect abnormal sounds and abnormal vibrations. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-292188 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the method disclosed in Patent Document 1, measurement conditions are selected based on data corresponding to predetermined measurement conditions, and therefore measurement conditions can only be searched for from among predetermined patterns of measurement conditions.

[0006] Therefore, there is a risk that the selected measurement conditions will not be optimal. To address this issue, it is possible to subdivide the measurement conditions more finely and increase the number of patterns. In that case, however, measurements would be performed under all patterns of measurement conditions, and the results would be evaluated to select the measurement conditions. This would require manpower and time.

[0007] In view of this, there has been a demand for an inspection control device that can quickly operate the powertrain under inspection into an operating state suitable for the inspection. [Means for solving the problem]

[0008] The means for solving the above problems and their effects will be described below. An inspection control device for solving the above problem is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in a powertrain of a vehicle based on data measuring sound or vibration emitted by the powertrain, and controls the powertrain. The inspection control device includes a storage device that stores a trained model that selects and outputs an action to manipulate a control amount of the powertrain when a state variable indicating an operating state of the powertrain is input, and a processing circuit that manipulates the control amount based on the action output by inputting the state variable into the trained model, and controls the powertrain. The trained model is a model trained by reinforcement learning that provides a reward so that the profit increases the fewer actions are required to shift the powertrain to an operating state in which the sound or vibration emitted from the powertrain is suitable for determining the presence or absence of an abnormality.

[0009] The inspection control device having the above configuration can transition the powertrain to an operating state suitable for inspection by controlling the powertrain using a trained model learned by reinforcement learning.

[0010] The trained model optimizes the policy for selecting an action so that the powertrain can be shifted with fewer actions to a driving state suitable for determining whether or not there is an abnormality in sound or vibration. Therefore, compared to a case where optimal conditions are searched for from among preset patterns of measurement conditions, a more diverse search can be performed. In addition, the search and optimization of the policy can be performed without relying on human labor.

[0011] According to the above-mentioned inspection control device, which operates the powertrain using a learned model optimized by reinforcement learning, the powertrain can be quickly brought into an operating state suitable for inspection.

[0012] One aspect of the inspection control device is applied to the inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on sound data measured by a microphone. In this inspection control device, the trained model is a model trained by reinforcement learning that provides a reward so that the profit increases the more the powertrain is brought into a driving state in which the sound caused by the abnormality to be determined is maximized with fewer actions.

[0013] In the case of an inspection control device applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality based on sound, as configured above, a trained model trained by reinforcement learning that provides rewards so that the profit increases as the driving state in which the sound caused by the abnormality to be determined is maximized with fewer actions can be shifted, thereby realizing an inspection control device that can quickly shift the powertrain to a driving state suitable for inspection.

[0014] One aspect of the inspection control device is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on sound data measured by a microphone. In this inspection control device, the trained model is a model trained by reinforcement learning that provides a reward so that the profit increases the more the powertrain is brought to a driving state that maximizes the difference between the sound emitted by the powertrain to be inspected and the sound emitted by a normal powertrain with fewer actions.

[0015] By learning using difference data obtained by removing sounds emitted by a normal powertrain from sounds emitted by the powertrain under test, learning can be performed while eliminating the influence of sounds other than sounds caused by abnormalities. Therefore, a trained model with higher accuracy can be obtained. According to the above configuration, the inspection control device equipped with a trained model trained using difference data obtained by removing sounds emitted by a normal powertrain from sounds emitted by the powertrain under test can more accurately transition the powertrain to an operating state suitable for inspection.

[0016] One aspect of the inspection control device is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on vibration data measured by a vibration sensor. In this inspection control device, the trained model is a model trained by reinforcement learning that provides a reward so that the profit increases the more the powertrain is shifted to a driving state in which vibration caused by the abnormality to be determined is maximized with fewer actions.

[0017] In the case of an inspection control device applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality based on vibration, as configured above, a trained model trained by reinforcement learning that provides rewards so that the profit increases as the driving state in which the vibration is maximized is shifted with fewer actions, can be mounted. This makes it possible to realize an inspection control device that can quickly shift the powertrain to a driving state suitable for inspection.

[0018] One aspect of the inspection control device is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on vibration data measured by a vibration sensor. In this inspection control device, the trained model is a model trained by reinforcement learning that provides a reward so that the profit increases the more the powertrain is brought to a driving state where the difference between the vibration emitted by the powertrain under inspection and the vibration emitted by a normal powertrain is maximized with fewer actions.

[0019] By learning using difference data obtained by subtracting vibrations emitted by a normal powertrain from vibrations emitted by the powertrain under test, learning can be performed while eliminating the effects of vibrations other than those caused by abnormalities. Therefore, a more accurate trained model can be obtained. According to the above configuration, the inspection control device equipped with a trained model trained using difference data obtained by subtracting vibrations emitted by a normal powertrain from vibrations emitted by the powertrain under test can more accurately transition the powertrain to an operating state suitable for inspection.

[0020] In one aspect of the inspection control device, the power train includes an internal combustion engine, and the state variables include oil pressure of the internal combustion engine. The level of oil pressure in an internal combustion engine affects the operation of the internal combustion engine. Therefore, the sound and vibration emitted by the powertrain also change depending on the oil pressure of the internal combustion engine. The above-mentioned inspection control device includes the oil pressure of the internal combustion engine in its state variables. Therefore, the above-mentioned inspection control device can select an action that reflects the influence of the oil pressure of the internal combustion engine.

[0021] In one aspect of the inspection control device, the power train includes an internal combustion engine, and the state variables include engine speed. The sound and vibration generated by the powertrain also change depending on the engine speed. The inspection control device includes the engine speed in its state variables. Therefore, the inspection control device can select an action that reflects the engine speed information.

[0022] In one aspect of the test control device, the power train includes an internal combustion engine, and the state variables include torque in the internal combustion engine. The sound and vibration generated by the powertrain also change depending on the magnitude of the torque in the internal combustion engine. The inspection control device described above includes the torque in the internal combustion engine in its state variables. Therefore, the inspection control device described above can select an action that reflects information about the torque in the internal combustion engine.

[0023] In one aspect of the inspection control device, the power train includes a transmission, and the state variables include a gear ratio. The sound and vibration generated by the powertrain also change depending on the gear ratio. The above-mentioned inspection control device includes the gear ratio in its state variables. Therefore, the above-mentioned inspection control device can select an action by reflecting the information on the gear ratio.

[0024] In one aspect of the inspection control device, the power train includes a transmission, and the state variables include an oil temperature of the transmission. The oil temperature of the transmission affects the operation of the transmission. Therefore, the sound and vibration generated by the powertrain also change depending on the oil temperature of the transmission. The above-mentioned inspection control device includes the oil temperature of the transmission in its state variables. Therefore, the above-mentioned inspection control device can select an action that reflects the influence of the oil temperature of the transmission.

[0025] In one aspect of the test control device, the power train includes a transmission, and the state variables include a torque in the transmission. The sound and vibration generated by the powertrain also change depending on the magnitude of the torque in the transmission. The inspection control device described above includes the torque in the transmission in its state variables. Therefore, the inspection control device described above can select an action that reflects the information on the torque in the transmission.

[0026] In one aspect of the inspection control device, the power train includes a transmission, and the action options include operating the transmission to increase the gear ratio and operating the transmission to decrease the gear ratio.

[0027] When the gear ratio is changed, the engine rotation speed, the load, the rotation speed of the output shaft of the transmission, etc. change, and the sound and vibration emitted from the powertrain change. The above-mentioned inspection control device includes, as a choice of actions, an action of operating the transmission to increase the gear ratio and an action of operating the transmission to decrease the gear ratio. Therefore, the above-mentioned inspection control device can control the powertrain to a driving state suitable for the inspection by operating the transmission according to the state of the powertrain.

[0028] In one aspect of the inspection control device, the powertrain includes an internal combustion engine, and the action options include operating the internal combustion engine to increase engine speed and operating the internal combustion engine to decrease engine speed.

[0029] When the engine speed is changed, the sound and vibration emitted from the powertrain are changed. The above-mentioned inspection control device includes, as a choice of actions, an action of operating the internal combustion engine so as to increase the engine speed and an action of operating the internal combustion engine so as to decrease the engine speed. Therefore, the above-mentioned inspection control device can control the powertrain to an operating state suitable for inspection by changing the engine speed according to the state of the powertrain.

[0030] In one aspect of the inspection control device, the power train includes an internal combustion engine, and the action options include operating the internal combustion engine to increase torque in the internal combustion engine and operating the internal combustion engine to decrease torque in the internal combustion engine.

[0031] When the torque in the internal combustion engine is changed, the sound and vibration emitted from the powertrain are changed. The above-mentioned inspection control device includes, as a choice of actions, an action of operating the internal combustion engine so as to increase the torque in the internal combustion engine and an action of operating the internal combustion engine so as to decrease the torque in the internal combustion engine. Therefore, the above-mentioned inspection control device can control the powertrain to an operating state suitable for inspection by changing the torque in the internal combustion engine according to the state of the powertrain.

[0032] In one embodiment of the inspection control device, the trained model is a neural network consisting of an input layer having a plurality of nodes to which the state variables are input, an intermediate layer having a plurality of nodes, and an output layer having a plurality of nodes that output Q values ​​for each action.

[0033] If the model uses reinforcement learning to calculate the Q value of each action using a neural network, it is easy to generate a model that can calculate the Q value of each action even if the number of state variables is large. The above-mentioned inspection control device is equipped with a trained model that uses a neural network to calculate the Q value of each action. Therefore, the above-mentioned inspection control device can input multiple state variables and can select appropriate actions in response to various states. [Brief description of the drawings]

[0034] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of an inspection system including an inspection control device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a sensor connected to an inspection control device. [Diagram 3] FIG. 3 is a flowchart showing the flow of a series of processes related to learning. [Figure 4] FIG. 4 is a schematic diagram illustrating the structure of a neural network. [Diagram 5] FIG. 5 is a flowchart showing a series of processing steps in which the inspection control device operates the powertrain. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Hereinafter, one embodiment of an inspection control device will be described with reference to FIGS. <Inspection system configuration> 1 shows the configuration of an inspection system equipped with an inspection control device 100. This inspection system controls a vehicle's powertrain to perform an inspection to determine the presence or absence of an abnormality based on the sound emitted by the powertrain. Specifically, the inspection control device 100 controls the powertrain to be in an operating state suitable for the inspection.

[0036] <Powertrain configuration> 1, the power train to be inspected is made up of an internal combustion engine 200 and a transmission 300. The inspection control device 100 controls the internal combustion engine 200 and the transmission 300. The inspection control device 100 also controls a dynamometer 400.

[0037] <Configuration of transmission 300> The transmission 300 includes a primary pulley 330, a secondary pulley 340, and a belt 350 wound around the primary pulley 330 and the secondary pulley 340. The transmission 300 transmits driving force via the belt 350. In the transmission 300, the winding radius of the belt 350 around the primary pulley 330 and the secondary pulley 340 changes by operating the primary pulley 330 and the secondary pulley 340. In other words, the transmission 300 is a continuously variable transmission that changes the gear ratio steplessly by changing the winding radius of the belt 350.

[0038] The transmission 300 includes a switching mechanism 320. The primary pulley 330 is connected to the output shaft 201 of the internal combustion engine 200 via the switching mechanism 320 and a torque converter 310. The switching mechanism 320 is a mechanism that switches between a reverse state in which the rotation of the output shaft 201 is reversed and transmitted to the primary pulley 330, and a forward state in which the rotation of the output shaft 201 is not reversed and transmitted to the primary pulley 330. The rotation of the output shaft 201 of the internal combustion engine 200 is transmitted to the primary pulley 330 via the switching mechanism 320. The rotation of the primary pulley 330 is transmitted to the secondary pulley 340 via a belt 350.

[0039] The transmission 300 includes a speed reduction mechanism 370 that reduces the rotation speed of the secondary pulley 340. An input shaft 371 of the speed reduction mechanism 370 is connected to the secondary pulley 340. A drive shaft 380, which is an output shaft of the speed reduction mechanism 370, is connected to a dynamometer 400. The drive shaft 380 is connected to a drive wheel when the powertrain is mounted on a vehicle.

[0040] The transmission 300 includes a hydraulic circuit 360. The hydraulic circuit 360 controls the hydraulic pressure in the primary pulley 330 and the hydraulic pressure in the secondary pulley 340. In this transmission 300, the winding radius of the belt 350 around the primary pulley 330 and the secondary pulley 340 is changed by changing the hydraulic pressure. In addition, the hydraulic circuit 360 switches between the forward state and the reverse state by controlling the hydraulic pressure supplied to the switching mechanism 320.

[0041] <Configuration of inspection control device 100> As shown in Fig. 2, the inspection control device 100 includes a processing circuit 110, a ROM 120, and a storage device 130 which is an electrically rewritable non-volatile memory. The ROM 120 stores a control program to be executed by the processing circuit 110. The inspection control device 100 operates the transmission 300 through control of the hydraulic circuit 360, switches between a forward state and a reverse state, and changes the gear ratio. The inspection control device 100 also controls each part of the internal combustion engine 200, thereby controlling the internal combustion engine 200. The inspection control device 100 is connected to the internal combustion engine 200, the transmission 300, and a sensor group 500 consisting of various sensors that detect the state of the vehicle.

[0042] For example, as shown in FIG. 2, a crank position sensor 501 is connected to the inspection control device 100. The crank position sensor 501 outputs a crank angle signal according to the rotation angle of a crankshaft, which is the output shaft 201 of the internal combustion engine 200. The inspection control device 100 calculates an engine rotation speed, which is the rotation speed of the output shaft 201, based on the crank angle signal output from the crank position sensor 501. An oil pressure sensor 502 is connected to the inspection control device 100. The oil pressure sensor 502 detects an oil pressure, which is the pressure of oil supplied to the internal combustion engine 200. A torque sensor 503 is connected to the inspection control device 100. The torque sensor 503 detects the torque of the output shaft 201 of the internal combustion engine 200. An oil temperature sensor 504 is connected to the inspection control device 100. The oil temperature sensor 504 detects an oil temperature, which is the temperature of the oil in the transmission 300. A microphone 505 is connected to the inspection control device 100.

[0043] The inspection system operates the powertrain to bring it into an operating state suitable for the inspection in order to perform an inspection to determine whether or not an abnormality has occurred in the transmission 300 based on the sound emitted from the transmission 300. Then, while the inspection system is used to maintain the powertrain in an operating state suitable for the inspection, an operator performing the inspection measures the sound emitted from the transmission 300 in more detail and determines whether or not an abnormality has occurred.

[0044] The microphone 505 measures sounds emitted from the transmission 300 in order to evaluate whether the driving conditions are suitable for the inspection. For this reason, the microphone 505 is installed facing the transmission 300.

[0045] The inspection control device 100 is also connected to the dynamometer 400 so that they can communicate with each other. The inspection control device 100 also controls the dynamometer 400. The inspection control device 100 obtains the torque and rotation speed of the drive shaft 380 from the dynamometer 400.

[0046] <Control to transition the powertrain to a state suitable for inspection> Next, the control executed by the inspection control device 100 to transition the powertrain to an operating state suitable for inspection will be described.

[0047] Data of a model used for this control is stored in the storage device 130 of the inspection control device 100. This model is a trained model trained by reinforcement learning. The inspection control device 100 controls the powertrain under inspection while being connected to the powertrain.

[0048] In this control, state variables including the engine rotation speed, the torque of the output shaft 201, the rotation speed of the drive shaft 380, etc. are input to the trained model to determine an action to manipulate the control amount of the powertrain. Then, the control amount of the powertrain is manipulated according to the determined action, thereby transitioning the powertrain to an operating state suitable for inspection with fewer actions. In other words, the trained model is a model that outputs an action to manipulate the control amount of the powertrain when a state variable is input.

[0049] Specifically, the trained model selects and outputs an action from the following six options: -Increase engine speed by a certain amount.

[0050] - Reduce engine speed by a certain amount. The torque of the output shaft 201 is increased by a certain amount. The torque of the output shaft 201 is reduced by a certain amount.

[0051] Increase the gear ratio by a certain amount. -Reduce the gear ratio by a certain amount. The processing circuit 110 of the inspection control device 100 controls the powertrain according to the action output by the trained model. For example, when an action to increase the engine rotation speed by a certain amount is selected and output by the trained model, the processing circuit 110 increases the target value of the engine rotation speed by a certain amount and controls the internal combustion engine 200. In addition, when an action to decrease the gear ratio is selected and output by the trained model, the processing circuit 110 decreases the target value of the gear ratio by a certain amount and controls the transmission 300.

[0052] <Learning pre-trained models> Next, learning of the trained model will be described. As described above, the trained model stored in the storage device 130 is trained by reinforcement learning.

[0053] 3 is a flowchart showing a series of processes involved in learning. In learning to generate a trained model stored in the storage device 130, the inspection control device 100, which is an agent, is made to select an action based on state variables. Then, the inspection control device 100 is made to execute the action thus selected. Then, the inspection control device 100 grants a reward according to the state after the selected action is executed.

[0054] As shown in Figure 3, in reinforcement learning, the process of acquiring a state, deciding an action, and giving a reward is repeated in this manner. This allows data to be collected on the state before the action, the selected action, and the state and reward after the action is executed. Using the data accumulated in this way, the model is optimized so that the optimal action can be selected to achieve the final goal based on the acquired state.

[0055] In this inspection control device 100, an action is selected based on state variables using a Q value. The Q value is an index value that represents the value of an action, and indicates the likelihood that a certain action will be selected in a certain state. The Q value is updated by the following formula (1).

[0056]

number

[0057] Q(s,a) is the expected profit that will be obtained in the future when action a is selected in state s. The reward is r. The subscript t in state s, action a, and reward r in the above formula (1) is the trial number indicating one step in the trial process repeated in time series. When state s changes after the action decision, the trial number t is incremented by one. In the following, the subscript is written with "_" followed by "_". Therefore, the reward r_t+1 in formula (1) is the reward obtained when action a_t is selected in state s_t and state s becomes state s_t+1. In formula (1), α is the learning rate and γ is the discount rate. Both the learning rate α and the discount rate γ are values ​​greater than 0 and less than 1. In addition, maxQ(s_t+1,a) is the Q value maximized by selecting action a. In this section, action a refers to the action that maximizes Q(s_t+1, a_t+1) among the actions a_t+1 that can be taken in state s_t+1.

[0058] As described above, the action a in the inspection control device 100 can be selected from six options related to the change of engine speed, the change of torque, and the change of gear ratio. Of course, this is just one example, and the content of the action a does not need to be limited to these. The options for the action a may be more or less than this, for example, an option to maintain each of the change of engine speed, the change of torque, and the change of gear ratio may be added, and the action a may be selected from nine options. Also, any of the change of engine speed, the change of torque, and the change of gear ratio may be omitted, or an operation of another part may be added to the action.

[0059] In the reinforcement learning of this embodiment, the reward r is given so that the profit increases as the powertrain is shifted to a driving state in which the sound emitted from the powertrain is suitable for determining whether or not there is an abnormality with fewer actions. Specifically, when it is determined that the sound pressure of the sound measured by the microphone 505 is maximized, the goal is reached and the reward r of "100" is given. At this time, the episode of trials for learning is terminated. Also, when it is not determined that the sound pressure is maximized, the trial is continued and a negative reward r, for example, "-1", is given each time an action is taken. Then, when the number of trials reaches a preset number even if the goal is not reached, the episode is terminated. For example, the preset number is 20 times.

[0060] The sound pressure is determined to be at its maximum when the number of trials is equal to or greater than a lower limit number that is smaller than the above-mentioned predetermined number, and the amount of change in sound pressure from the sound pressure immediately before the trial is less than a certain level continues for a predetermined number of times, for example, three times. In other words, the sound pressure is determined to be at its maximum when it is estimated that the change in sound pressure has converged and the sound pressure will not increase much even if the trial is repeated.

[0061] When an action a_t is selected in a state s_t, the next state s_t+1 can be specified by selecting and executing an action a and obtaining the state variables. In this embodiment, the engine rotation speed, the oil pressure detected by the oil pressure sensor 502, the torque of the output shaft 201 detected by the torque sensor 503, and the oil temperature detected by the oil temperature sensor 504 are input to the model as state variables. Also, in this embodiment, the rotation speed and torque of the drive shaft 380 acquired from the dynamometer 400 are also input to the model as state variables.

[0062] Information indicating variables and functions referenced in the learning process is stored in the storage device 130. The inspection control device 100 observes state variables, determines an action a according to the observed state variables, and evaluates the state after the action a is executed. Then, the inspection control device 100 gives a reward r and updates the model that calculates the Q value.

[0063] To this end, in the learning process, the inspection control device 100 sequentially stores data on the state before the action, the selected action, and the state and reward after the action is performed in the memory device 130, as described above.

[0064] In this embodiment, DQN (Deep Q-Network), which is a method for approximately calculating a Q value, is adopted. In DQN, a multi-layer neural network is used to estimate the Q value. In this embodiment, a neural network is adopted that receives a state s as input and outputs a value of Q(s, a), which is a Q value corresponding to the number of selectable actions a.

[0065] Fig. 4 is a schematic diagram showing a neural network that outputs a Q value. In Fig. 4, the neural network receives m values ​​X, which are the above state variables, as input, and outputs n Q values ​​corresponding to n actions a. In Fig. 4, the m input values ​​in trial number t are shown as X_1 to X_m.

[0066] As described above, in this embodiment, the engine speed, the oil pressure of the internal combustion engine 200, the torque of the output shaft 201, the oil temperature of the transmission 300, and the rotation speed and torque of the drive shaft 380 are input as state variables. In other words, m is "6".

[0067] Here, n is the number of selectable actions a, and the output of the neural network is a Q value when a specific action a is selected in the input state s. In FIG. 4, the Q values ​​of the selectable actions a_1t to a_nt in the trial number t are shown as Q(s_t, a_1t) to Q(s_t, a_nt). The "s_t" written in this Q value is a character representing the state s input in the trial number t, that is, X_1 to X_m. In this embodiment, the number of selectable actions a is six in total, since there are two ways each for the change of engine speed, the change of torque, and the change of gear ratio. Therefore, n is "6".

[0068] The neural network shown in Figure 4 has an input layer, multiple intermediate layers, and an output layer. This neural network is a multi-layered neural network that performs a calculation in which each node in each layer multiplies the input of the previous layer by a weight w and adds a bias b to obtain an output that has passed through an activation function. Note that in Figure 4, the transmission lines connecting the nodes of adjacent layers are omitted.

[0069] The structure of the neural network is specified by information such as the weight w and bias b in each layer, the activation function, and the order of the layers. Therefore, in the inspection control device 100, parameters for specifying this multi-layer neural network are recorded in the storage device 130. During learning, the weight w, which is a variable value in the neural network, is updated.

[0070] Next, the procedure of the learning process will be described in more detail with reference to the flowchart shown in FIG. As shown in FIG. 3, when learning starts, the processing circuit 110 sets an initial value in the processing of step S100. Specifically, the processing circuit 110 sets the trial number t to the initial value "0". Then, the powertrain is operated to the initial state, and the trial starts from there. It should be noted that the number of episodes is "0" immediately after the start of learning. If a negative judgment is made in the processing of step S180 described later and step S100 is executed for the second or subsequent time, the number of episodes is incremented by one in the processing of step S100.

[0071] In the next step S110, the processing circuit 110 acquires the state s of the power train. Specifically, the processing circuit 110 acquires the above-mentioned state variables. That is, the engine speed, the oil pressure of the internal combustion engine 200, the torque of the output shaft 201, the oil temperature of the transmission 300, and the rotation speed and torque of the drive shaft 380 are acquired.

[0072] Next, in the process of step S120, the processing circuit 110 determines the action a. Specifically, the processing circuit 110 determines the action a using the ε greedy algorithm while inputting state variables to the neural network and calculating the Q value.

[0073] In the ε-greedy method, action a with the largest Q-value is selected, but action a is determined randomly at a certain rate. When learning first begins, action a is determined randomly with little regard for the Q-value. Then, as learning progresses, that is, as the number of episodes increases, the rate at which action a is determined randomly is gradually reduced. Eventually, action a is selected based only on the Q-value.

[0074] When the action is determined in the process of step S120, the processing circuit 110 controls the powertrain in accordance with the determined action a. Then, the processing circuit 110 advances the process to step S130.

[0075] In the process of step S130, the processing circuit 110 acquires the state s after the operation according to the action a. That is, the same process as the process of step S110 is performed to acquire the state variable. Note that, when the current trial number is t and the selected action a is action a_t, the state s acquired in the process of step S130 is state s_t+1.

[0076] Next, the processing circuit 110 grants a reward r in the process of step S140. Specifically, as described above, when the processing circuit 110 determines that the sound pressure of the sound measured by the microphone 505 has reached a maximum, the processing circuit 110 grants a reward r of "100" as the goal has been reached. On the other hand, when it is not determined that the sound pressure has reached a maximum, the processing circuit 110 grants a reward r of "-1". After granting the reward r in this way, the processing circuit 110 advances the process to step S150.

[0077] In the process of step S150, the processing circuit 110 saves data. Specifically, the processing circuit 110 sequentially stores data of a state s_t before an action, a selected action a_t, and a state s_t+1 and a reward r_t+1 after the action a_t is executed in the storage device 130. Then, the processing circuit 110 advances the process to step S160.

[0078] In the process of step S160, the processing circuit 110 judges whether or not the goal condition is satisfied. The goal condition is a logical sum of the following two conditions. That is, the processing circuit 110 judges that the goal condition is satisfied if at least one of them is satisfied.

[0079] In the process of step S140, the reward r was given as “100”. Attempt number t has reached the set number of attempts. When the processing circuit 110 determines in the process of step S160 that the goal condition is not satisfied (step S160: NO), the processing circuit 110 returns to step S110 and executes the processes of steps S110 to S160 again. When the processing circuit 110 returns to step S110, the processing circuit 110 increments the trial number t by one.

[0080] On the other hand, if it is determined in the process of step S160 that the goal condition is met (step S160: YES), the processing circuit 110 advances the process to step S170. In the process of step S170, the processing circuit 110 executes learning.

[0081] In this learning process, the weights w of the neural network are updated and optimized. The reason for updating the Q value according to the Q value update formula shown in the above formula (1) is to bring it closer to the relationship expressed by the following formula (2).

[0082]

number

[0083] Therefore, the weights w of the neural network must be updated so that Q(s_t, a_t) output from the output layer approaches the action value shown on the right side of the above formula (2). Therefore, in this embodiment, the error function E is defined as the following formula (3), and the weights w of the neural network are updated by backpropagation so that the error function E becomes small.

[0084]

number

[0085] Here, the value of maxQ(s_t+1, a) needs to be calculated by inputting the state s_t+1 to the neural network. Therefore, in this embodiment, the processing circuit 110 creates a mini-batch by repeating trials and accumulating a plurality of data in the storage device 130. Then, the processing circuit 110 executes learning using the mini-batch (Experience Replay). Furthermore, the processing circuit 110 calculates the error function E described above based on past data, targeting the action value of past actions, and updates the neural network (Fixed Target Q-Network).

[0086] After updating the weight w of the neural network through the learning in step S170, the processing circuit 110 advances the process to step S180. Then, in the process of step S180, the processing circuit 110 determines whether the number of episodes is equal to or greater than a predetermined threshold. The threshold is a threshold for determining the completion of learning. The magnitude of the threshold is a predetermined hyperparameter such that learning is completed when the number of episodes is equal to or greater than the threshold. In this embodiment, the threshold is, for example, "1000".

[0087] When it is determined in the process of step S180 that the number of episodes is less than the threshold value (step S180: NO), the processing circuit 110 returns the process to step S100. At this time, the processing circuit 110 increments the number of episodes by one and starts a new episode.

[0088] On the other hand, if it is determined in the process of step S180 that the number of episodes is equal to or greater than the threshold (step S180: YES), the processing circuit 110 proceeds to the process of step S190. In this case, optimization of the neural network is completed, and learning is completed. Therefore, the processing circuit 110 stores the parameters of the neural network, which is the model that has been trained in the process of step S190, in the storage device 130.

[0089] After storing the information about the trained model in the memory device 130, the processing circuit 110 ends the series of routines related to learning. Thus, in this embodiment, when the processing circuit 110 determines that the sound pressure of the measured sound has reached its maximum, it assigns a reward r of "100" and ends the episode. On the other hand, when the processing circuit 110 determines that the sound pressure has not reached its maximum, it assigns a reward r of "-1". By the processing circuit 110 assigning the reward r in this manner, the profit, which is the sum of the rewards r, increases as the driving state at which the sound pressure is maximized is shifted with fewer actions. Therefore, through learning, the neural network comes to output a Q value that selects an action a that quickly reaches a driving state at which the sound pressure is maximized according to the state s.

[0090] The inspection control device 100, which is equipped with a memory device 130 in which the learned model learned by reinforcement learning in this manner is stored, executes the routine shown in FIG. 5 to control the powertrain to be inspected so that it is in an operating state suitable for the inspection.

[0091] 5, when this routine starts, the processing circuit 110 executes the process of step S300. In the process of step S300, the processing circuit 110 acquires the state s in the same manner as the process of step S110 described with reference to FIG.

[0092] Next, in the process of step S310, the processing circuit 110 inputs the state variables acquired through the process of step S300 into the trained neural network and determines the action a. At this time, the action a corresponding to the largest Q value output by the neural network is selected. After determining the action a in this way, the processing circuit 110 advances the process to step S320.

[0093] In the process of step S320, the processing circuit 110 executes the selected action a to control the powertrain, and then ends this series of processes. The inspection control device 100 repeatedly executes this routine. In this way, the powertrain is repeatedly operated by selecting an action a based on a state s using a learned model. In this way, the powertrain is operated toward an operating state suitable for the inspection.

[0094] <Action of this embodiment> According to the above configuration, the powertrain can be controlled by selecting the action a that maximizes the Q value. The neural network that outputs the Q value is optimized as a result of repeated trials through the above-mentioned learning. The neural network is trained by reinforcement learning that gives rewards so that the fewer actions required to transition to a driving state in which the sound emitted from the powertrain is suitable for testing, the greater the profit. Therefore, according to this inspection control device 100, the powertrain can be quickly transitioned to a driving state suitable for testing.

[0095] <Effects of this embodiment> (1) The inspection control device 100 can transition the powertrain to an operating state suitable for inspection by controlling the powertrain using a trained model learned by reinforcement learning.

[0096] (2) The trained model optimizes the policy for selecting action a so that the powertrain can be shifted with fewer actions to a driving state suitable for determining whether the sound is abnormal. Therefore, compared to searching for optimal conditions from among preset patterns of measurement conditions, it is possible to perform a more diverse search. In addition, the search and optimization of the policy can be performed without relying on human labor.

[0097] In this way, the inspection control device 100 that operates the powertrain using a learned model optimized by reinforcement learning can quickly transition the powertrain to an operating state suitable for inspection.

[0098] (3) The level of the oil pressure of the internal combustion engine 200 affects the operation of the internal combustion engine 200. Therefore, the sound and vibration generated by the powertrain also change depending on the oil pressure of the internal combustion engine 200. The inspection control device 100 includes the oil pressure of the internal combustion engine 200 in the state variables. Therefore, the inspection control device 100 can select the action a by reflecting the influence of the oil pressure of the internal combustion engine 200.

[0099] (4) The sound and vibration generated by the powertrain also change depending on the engine speed. The inspection control device 100 includes the engine speed in its state variables. Therefore, the inspection control device 100 can select the action a by reflecting the engine speed information.

[0100] (5) The sound and vibration generated by the powertrain also change depending on the magnitude of the torque in the internal combustion engine 200. The inspection control device 100 includes the torque in the internal combustion engine 200 in the state variables. Therefore, the inspection control device 100 can select the action a by reflecting the information on the torque in the internal combustion engine 200.

[0101] (6) The sound and vibration generated by the powertrain also change depending on the gear ratio. The inspection control device 100 includes the gear ratio in its state variables. Therefore, the inspection control device 100 can select the action a by reflecting the information on the gear ratio.

[0102] (7) The oil temperature of the transmission 300 affects the operation of the transmission 300. Therefore, the sound and vibration generated by the powertrain also change depending on the oil temperature of the transmission 300. The inspection control device 100 includes the oil temperature of the transmission 300 in the state variables. Therefore, the inspection control device 100 can select the action a by reflecting the influence of the oil temperature of the transmission 300.

[0103] (8) The sound and vibration generated by the powertrain also change depending on the magnitude of the torque in the transmission 300. The inspection control device 100 includes the torque in the transmission 300 in the state variables. Therefore, the inspection control device 100 can select the action a by reflecting the information on the torque in the transmission 300.

[0104] (9) When the gear ratio is changed, the engine rotation speed, the load, the rotation speed of the drive shaft 380 which is the output shaft of the transmission 300, and the like change, and the sound and vibration emitted from the powertrain change. The options for the action a in the inspection control device 100 include an action of operating the transmission 300 so as to increase the gear ratio, and an action of operating the transmission 300 so as to decrease the gear ratio. Therefore, the inspection control device 100 can control the powertrain to a driving state suitable for the inspection by operating the transmission 300 in accordance with the state s of the powertrain.

[0105] (10) When the engine rotation speed is changed, the sound and vibration emitted from the powertrain change. The inspection control device 100 includes, in the options for action a, an action of operating the internal combustion engine 200 so as to increase the engine rotation speed, and an action of operating the internal combustion engine 200 so as to decrease the engine rotation speed. Therefore, the inspection control device 100 can control the powertrain to an operating state suitable for the inspection by changing the engine rotation speed according to the state s of the powertrain.

[0106] (11) When the torque in the internal combustion engine 200 is changed, the sound and vibration emitted from the powertrain change. The inspection control device 100 includes, in the options for action a, an action of operating the internal combustion engine 200 so as to increase the torque in the internal combustion engine 200, and an action of operating the internal combustion engine 200 so as to decrease the torque in the internal combustion engine 200. Therefore, the inspection control device 100 can control the powertrain to an operating state suitable for inspection by changing the torque in the internal combustion engine 200 according to the state s of the powertrain.

[0107] (12) A model that uses reinforcement learning to calculate the Q value of each action using a neural network, such as the inspection control device 100, can easily generate a model that calculates the Q value of each action even if there are a large number of state variables. The inspection control device 100 equipped with such a neural network model can input multiple state variables and can select an appropriate action a in response to various states.

[0108] This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction. In the above embodiment, an example has been shown in which the microphone 505 is provided to measure the sound emitted by the transmission 300, and the presence or absence of an abnormality in the transmission 300 is determined. However, a similar configuration may also be adopted for an inspection control device that is applied to an inspection system in which the microphone 505 is provided to measure the sound emitted by the internal combustion engine 200, and the presence or absence of an abnormality in the internal combustion engine 200 is determined.

[0109] In the above embodiment, an example of the inspection control device 100 is shown, which performs an inspection to measure sound using the microphone 505 and determine the presence or absence of an abnormality based on the sound data. However, a configuration similar to that of the inspection control device 100 can also be applied to an inspection control device that performs an inspection to determine the presence or absence of an abnormality based on vibrations generated by the powertrain.

[0110] In this case, the vibration sensor 506 is connected to the inspection control device 100 as shown by the dashed line in Fig. 2. Then, the inspection control device 100 performs an inspection to determine the presence or absence of an abnormality based on the vibration data measured by the vibration sensor 506. In this case, for example, the processing circuit 110 gives "100" as the reward r when it is determined that the amplitude of the vibration measured by the vibration sensor 506 has reached a maximum. On the other hand, the processing circuit 110 gives "-1" as the reward r when it is not determined that the amplitude of the vibration measured by the vibration sensor 506 has reached a maximum.

[0111] That is, the inspection control device 100 in this case is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in a powertrain based on vibration data measured by the vibration sensor 506. The inspection control device 100 is equipped with a trained model that has been trained by reinforcement learning, which provides a reward so that the profit increases the more the powertrain is shifted to a driving state in which the vibration caused by the abnormality to be determined is maximized with fewer actions.

[0112] According to the inspection control device 100 equipped with such a learned model, as in the above embodiment, the powertrain can be quickly brought into an operating state suitable for inspection. Sound and vibration data may be used to determine the intensity of specific components obtained by frequency analysis. For example, sound and vibration data may be frequency analyzed using fast Fourier transform or wavelet transform. Then, based on the intensity of the components specific to the abnormality of the object of inspection, it may be determined whether the operating state is suitable for the inspection.

[0113] Alternatively, the difference between the sound emitted by the powertrain to be judged and the sound emitted by a normal powertrain may be used. In this case, the processing circuit 110 evaluates the driving state by using the difference between the sound emitted by the powertrain to be judged and the sound emitted by a normal powertrain as an index value. The trained model is trained by reinforcement learning that provides rewards so that the profit increases the more the powertrain is shifted to the driving state where this difference is maximized with fewer actions.

[0114] The data on the sounds generated by a normal powertrain can be obtained in advance by operating the normal powertrain under various driving conditions and recording the sounds. Learning using data of the difference between the sound emitted by the powertrain to be judged and the sound emitted by a normal powertrain, can eliminate the influence of sounds other than those caused by abnormalities. Therefore, a more accurate trained model can be obtained. Therefore, the powertrain can be more accurately shifted to an operating state suitable for the inspection. The same can be said about the inspection control device 100 applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on data of vibrations measured by the vibration sensor 506. In this case, the processing circuit 110 evaluates the operating state by using the difference between the vibrations emitted by the powertrain to be inspected and the vibrations emitted by the normal powertrain as an index value. The trained model is trained by reinforcement learning that gives a reward so that the profit increases the more the powertrain is shifted to an operating state where this difference is maximized with fewer actions. Learning using data of the difference between the vibrations emitted by the powertrain to be inspected and the vibrations emitted by the normal powertrain, can eliminate the influence of vibrations other than those caused by abnormalities. Therefore, a more accurate trained model can be obtained.

[0115] The types and number of state variables can be changed as appropriate. For example, some of the state variables exemplified in the above embodiment can be omitted. Also, the number of types of state variables can be increased. The number and types of options for the action a can be changed as appropriate. For example, in the inspection of the internal combustion engine 200, the change of the gear ratio may be omitted from the options for the action a, and only the engine speed and the torque of the internal combustion engine 200 may be changed.

[0116] The evaluation of whether the driving state is suitable for inspection is not limited to the magnitude of sound or vibration. For example, it is also possible to evaluate whether the driving state is suitable for inspection using a machine-learned model that outputs the probability that an abnormality to be determined occurs by inputting sound or vibration data. Note that such a machine-learned model may be, for example, a model that outputs the probability that an abnormality occurs that is learned by supervised learning. Using such a machine-learned model, it may be possible to determine whether the driving state is suitable for inspection through reinforcement learning.

[0117] In the above embodiment, an example was given in which a neural network was used to output a Q value. The model for calculating the Q value does not necessarily have to be a neural network. For example, when the number of state variables or the number of options for action a are small, action a can be determined using a Q table that stores the relationship between state variables and action a and the Q value in a table. In this case, a trained model can be generated by reinforcement learning using Q-learning that updates the Q value in the Q table according to the above formula (1) every time a trial is performed.

[0118] The transmission 300 is not limited to a continuously variable transmission. The same configuration as in the above embodiment may be applied to an inspection system for determining the presence or absence of an abnormality in a stepped transmission. The measurement data of state variables, sounds, and vibrations do not have to be the values ​​collected by sensors or the like. For example, data collected by sensors or the like that has been preprocessed may be used. Specifically, in order to remove noise contained in the collected data, a moving average value of the collected data may be calculated and used. Furthermore, maximum and minimum values ​​over a certain period of time may be used.

[0119] The inspection control device 100 is not limited to a device equipped with a processing circuit 110, a ROM 120, and a storage device 130 and executing software processing. For example, the inspection control device 100 may be equipped with a dedicated hardware circuit such as an ASIC that performs hardware processing of at least a part of what has been software-processed in the above embodiment. That is, the inspection control device 100 may have any of the following configurations (a) to (c). (a) A processing circuit that executes all of the above processing according to a program, and a program storage device such as a ROM that stores the program. (b) A processing circuit and a program storage device that execute a part of the above processing according to a program, and a dedicated hardware circuit that executes the remaining processing. (c) A dedicated hardware circuit that executes all of the above processing. Here, the software execution device equipped with the processing circuit and the program storage device, and the dedicated hardware circuit may be multiple. [Explanation of symbols]

[0120] 100...Inspection control device 110... Processing circuit 120...ROM 130...Storage device 200…Internal combustion engine 201...Output shaft 300…Gearbox 360…Hydraulic circuit 371…Input shaft 380…Drive shaft 400…Dynamometer 500…Sensor group 501...Crank position sensor 502...Rotational speed sensor 503...Torque sensor 504...Oil temperature sensor 505…Microphone 506...Vibration sensor

Claims

1. An inspection control device is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in a powertrain of a vehicle based on data obtained by measuring sounds or vibrations emitted by the powertrain, and controls the powertrain, a storage device that stores a learned model that selects and outputs an action for manipulating a control amount of the powertrain when a state variable indicating an operating state of the powertrain is input; A processing circuit that operates the control amount based on the action output by inputting the state variables into the learned model, and controls the powertrain; The trained model is a model trained by reinforcement learning in which a reward is given so that the profit increases the more the powertrain is shifted with fewer actions to a driving state in which the sound or vibration emitted from the powertrain is suitable for determining whether or not there is an abnormality. Inspection control device.

2. The present invention is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on sound data measured by a microphone, The trained model is a model trained by reinforcement learning that provides a reward so that the profit increases the less the powertrain is shifted to a driving state in which the sound caused by the abnormality to be determined is maximized. The inspection control device according to claim 1 .

3. The present invention is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on sound data measured by a microphone, The trained model is a model trained by reinforcement learning that provides a reward so that the profit increases the less the powertrain is transferred with fewer actions to a driving state in which the difference between the sound emitted by the powertrain under test and the sound emitted by a normal powertrain is maximized. The inspection control device according to claim 1 .

4. The present invention is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on vibration data measured by a vibration sensor, The trained model is a model trained by reinforcement learning that provides a reward so that the profit increases the less the powertrain is shifted to a driving state in which vibration caused by an abnormality to be determined is maximized. The inspection control device according to claim 1 .

5. The present invention is applied to an inspection system for performing an inspection to determine the presence or absence of an abnormality in the powertrain based on vibration data measured by a vibration sensor, The trained model is a model trained by reinforcement learning in which a reward is given so that the profit increases the less the powertrain is brought to a driving state in which the difference between the vibration emitted by the powertrain under test and the vibration emitted by a normal powertrain is maximized. The inspection control device according to claim 1 .

6. the power train includes an internal combustion engine; The state variables include the oil pressure of the internal combustion engine. The inspection control device according to any one of claims 1 to 5.

7. the power train includes an internal combustion engine; The state variables include the engine speed. The inspection control device according to any one of claims 1 to 6.

8. the power train includes an internal combustion engine; The state variables include a torque in the internal combustion engine. The inspection control device according to any one of claims 1 to 7.

9. The power train includes a transmission. The state variables include the gear ratio. The inspection control device according to any one of claims 1 to 8.

10. The power train includes a transmission. The state variables include the oil temperature of the transmission. The inspection control device according to any one of claims 1 to 9.

11. The power train includes a transmission. The state variables include the torque in the transmission. The inspection control device according to any one of claims 1 to 10.

12. The power train includes a transmission. The options for the behavior include a behavior of operating the transmission so as to increase the gear ratio, and a behavior of operating the transmission so as to decrease the gear ratio. The inspection control device according to any one of claims 1 to 11.

13. the power train includes an internal combustion engine; The action options include an action of operating the internal combustion engine so as to increase the engine rotation speed, and an action of operating the internal combustion engine so as to decrease the engine rotation speed. The inspection control device according to any one of claims 1 to 12.

14. the power train includes an internal combustion engine; The options for the actions include an action of operating the internal combustion engine so as to increase the torque in the internal combustion engine, and an action of operating the internal combustion engine so as to decrease the torque in the internal combustion engine. The inspection control device according to any one of claims 1 to 13.

15. The trained model is a neural network including an input layer having a plurality of nodes to which the state variables are input, an intermediate layer having a plurality of nodes, and an output layer having a plurality of nodes that output a Q value for each action. The inspection control device according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Knowledge preparation support system

    JP2008292188A

  • Production system for setting determination value of variable related to abnormality of product

    JP2017199074A

  • Controller and control method for automatic driving robot

    JP2020056737A

  • Vehicle control data generation method, vehicle control device, vehicle control system, on-vehicle device and vehicle learning device

    JP2021067191A