Signal processing device and signal processing method

Abstract

To perform learning so as to properly separate an input physical quantity into a noise component and an extraction signal.SOLUTION: A signal processing apparatus 10 includes: a noise-reduced input physical quantity generation unit 15 which generates a noise-reduced input physical quantity 90A obtained by reducing a noise component (noise 91b) included in an input physical quantity (engine peripheral sound 90); a selection unit 16 which selects one or both of the input physical quantity and the noise-reduced input physical quantity; and a learning unit 21 which learns a weight of a neural network for generating a noise reducing mask α for reducing the noise component, from the input physical quantity and / or the noise-reduced input physical quantity selected by the selection unit.SELECTED DRAWING: Figure 3A

Description

[Technical field]

[0001] The present invention relates to a signal processing device and a signal processing method that perform learning for effectively separating an input physical quantity into a noise component and an extracted signal from which the noise component has been removed. [Background technology]

[0002] For example, in an internal combustion engine such as a gasoline engine, the ignition timing is generally advanced as much as possible within a crank angle range where knocking does not occur, in order to improve output torque. Therefore, in the process of adjusting the ignition timing, a tester or a knocking determination device determines whether knocking occurs. An example of such a knocking determination device is described in Patent Document 1.

[0003] The knocking determination device described in Patent Document 1 selects a target signal to be compared with a determination signal for determining the presence or absence of knocking based on a condition determined by a relationship with the determination signal (for example, a relationship in terms of time or a relationship in terms of operating conditions).

[0004] The knocking determination device described in Patent Document 1 can only present the determination result of whether or not knocking occurs to the outside, and it is difficult to verify the determination result. Therefore, the inventors of the present invention have proposed the devices described in Patent Document 2 or Patent Document 3 as devices capable of estimating a desired physical quantity that can verify the determination result.

[0005] The device described in Patent Document 2 is described as "a learning device for removing noise components contained in an input physical quantity and estimating a desired physical quantity from the input physical quantity from which the noise components have been removed, wherein the noise components are noise other than a knocking sound generated in an internal combustion engine, the input physical quantity is the sound pressure of the internal combustion engine including the noise and the knocking sound, and the desired physical quantity is the in-cylinder pressure of the internal combustion engine when knocking occurs, and the learning device is characterized in that it includes a learning unit that removes the noise using a neural network and learns neural network weights that generate a mask that extracts the knocking sound, and / or a transfer function that converts the knocking sound extracted by the mask into the in-cylinder pressure of the internal combustion engine when knocking occurs."

[0006] The device described in Patent Document 3 is described as "a knocking determination device that removes noise other than the knocking sound generated in an internal combustion engine and estimates the knocking sound from which the noise has been removed, characterized in that the knocking determination device includes: a learning unit that uses a neural network to learn weights of a neural network that generates a mask that removes the noise and extracts the knocking sound, and a transfer function that converts the knocking sound extracted by the mask into the in-cylinder pressure of the internal combustion engine at the time when knocking occurs; and a second estimation unit that uses the mask by a neural network to estimate the knocking sound from which the noise has been removed from the knocking sound including the noise." [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2017-44148 A [Patent Document 2] Patent No. 6605170 [Patent Document 3] Patent No. 6651040 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the conventional techniques described in Patent Documents 2 and 3, when the amount of noise components (noise amount) included in the input physical quantity is large (particularly when the sound pressure of the noise is relatively larger than the signal to be taken from the input physical quantity (signal to be extracted)), the noise components after separation may contain components related to the teacher signal. Therefore, there are times when the input physical quantity cannot be satisfactorily separated into the noise components and the extraction signal.

[0009] The present invention has been made in order to solve the above-mentioned problems, and has as its main object to provide a signal processing device and a signal processing method that perform learning to satisfactorily separate an input physical quantity into a noise component and an extracted signal. [Means for solving the problem]

[0010] In order to achieve the above object, the present invention provides a signal processing device, comprising: a noise-reduced input physical quantity generating unit that generates a noise-reduced input physical quantity in which a noise component included in an input physical quantity has been reduced; a selecting unit that selects either the input physical quantity or the noise-reduced input physical quantity, or both; and a learning unit that learns weights of a neural network that generates a noise removal mask for removing the noise component from the input physical quantity and the noise-reduced input physical quantity selected by the selecting unit. the noise-reduced input physical quantity generation unit generates the noise-reduced input physical quantity by reducing the noise component from the input physical quantity using a processing mask obtained by filtering the noise removal mask, the selection unit supplies a data set obtained by shuffling the input physical quantity and the noise-reduced input physical quantity to the learning unit, and the learning unit learns weights of the neural network so that the correlation between the noise component and a teacher signal becomes weaker and the correlation between the teacher signal and an extracted signal obtained by removing the noise component from the input physical quantity or the noise-reduced input physical quantity becomes stronger. The composition is as follows.

[0011] The present invention also provides a signal processing method, comprising: a noise-reduced input physical quantity generating step of generating a noise-reduced input physical quantity in which a noise component contained in an input physical quantity has been reduced; a selection step of selecting either the input physical quantity or the noise-reduced input physical quantity, or both of the input physical quantity and the noise-reduced input physical quantity; and a learning step of learning weights of a neural network that generates a noise removal mask for removing the noise component from the input physical quantity and the noise-reduced input physical quantity selected in the selection step. In the noise-reduced input physical quantity generating step, a processing mask obtained by filtering the noise removal mask is used to generate the noise-reduced input physical quantity in which the noise component is reduced from the input physical quantity, and in the selection step, a data set in which the input physical quantity and the noise-reduced input physical quantity are shuffled is supplied to a learning unit that executes the processing of the learning step, and in the learning step, weights of the neural network are learned so that the correlation between the noise component and a teacher signal becomes small and the correlation between the teacher signal and an extracted signal obtained by removing the noise component from the input physical quantity or the noise-reduced input physical quantity becomes large. The composition is as follows. Other means will be described later. Effect of the Invention

[0012] According to the present invention, learning can be performed to effectively separate an input physical quantity into a noise component and an extracted signal. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing the overall configuration of a signal processing system including a signal processing device (estimation device) according to a first embodiment. [Diagram 2] 1 is a block diagram showing a configuration of a signal processing device (estimation device) according to a first embodiment. [Figure 3A] FIG. 13 is an explanatory diagram of a learning mode. [Figure 3B] FIG. 2 is an explanatory diagram of the operation of the signal processing device (estimation device) according to the first embodiment in a learning mode. [Figure 4A] FIG. 11 is an explanatory diagram of a threshold calculation mode. [Figure 4B] FIG. 2 is an explanatory diagram of the operation of the signal processing device (estimation device) according to the first embodiment in a threshold calculation mode. [Figure 5A] FIG. 11 is an explanatory diagram of a determination mode. [Figure 5B] FIG. 2 is an explanatory diagram of the operation of the signal processing device (estimation device) according to the first embodiment in a determination mode. [Figure 6A] FIG. 13 is an explanatory diagram of a separation mode. [Figure 6B] FIG. 2 is a diagram illustrating the operation of the signal processing device (estimation device) according to the first embodiment in a separation mode. [Figure 7A] FIG. 13 is an explanatory diagram of a sensory test mode. [Figure 7B] FIG. 2 is an explanatory diagram of the operation of the signal processing device (estimation device) according to the first embodiment in a sensory test mode. [Figure 8A] FIG. 4 is an explanatory diagram illustrating the relationship between a noise component and a teacher signal during learning. [Figure 8B] FIG. 4 is an explanatory diagram illustrating the relationship between an extracted signal and a teacher signal during learning. [Figure 8C] FIG. 4 is an explanatory diagram illustrating the relationship between an extracted signal and an estimated signal during learning. [Figure 9]FIG. 4 is an explanatory diagram of learning weights of a neural network that generates a noise removal mask in the first embodiment. [Figure 10A] FIG. 11 is an explanatory diagram of a calculation example in which phase components of pressure, vibration, sound, etc. are not taken into consideration. [Figure 10B] FIG. 11 is an explanatory diagram of a calculation example in which phase components of pressure, vibration, sound, etc. are taken into consideration. [Figure 11A] FIG. 1 is an explanatory diagram of signal processing in a sensory test (1). [Figure 11B] FIG. 2 is an explanatory diagram (2) of signal processing in a sensory test. [Figure 11C] FIG. 3 is an explanatory diagram of signal processing in a sensory test. [Figure 12A] FIG. 4 is an explanatory diagram of signal processing in a sensory test. [Figure 12B] FIG. 5 is an explanatory diagram (5) of signal processing in a sensory test. [Figure 12C] FIG. 6 is an explanatory diagram of signal processing in a sensory test. [Figure 13] 4 is a flowchart showing a data collection process of the signal processing device (estimation device) in the first embodiment. [Figure 14A] 4 is a flowchart showing a learning process of the signal processing device (estimation device) in the first embodiment. [Figure 14B] 13 is a flowchart showing a subroutine of a learning process. [Figure 14C] 13 is a flowchart showing a modified example of the learning process subroutine. [Figure 15A] 4 is a flowchart showing a threshold calculation process of the signal processing device (estimation device) in the first embodiment. [Figure 15B] 18 is a flowchart showing a threshold calculation process of the signal processing device (estimation device) performed after the separation process of FIG. 17 and the sensory test process of FIG. 18 in the first embodiment. [Figure 16] 4 is a flowchart showing a determination process of a signal processing device (estimation device) in the first embodiment. [Figure 17]5 is a flowchart showing a separation process of a signal processing device (estimation device) in the first embodiment. [Figure 18] 5 is a flowchart showing a sensory test process of the signal processing device (estimation device) in the first embodiment. [Figure 19A] FIG. 1 is a block diagram showing a configuration of a signal processing device (estimation device) according to a comparative example. [Figure 19B] FIG. 4 is an explanatory diagram of a learning mode of a signal processing device (estimation device) according to a comparative example. [Figure 19C] FIG. 10 is an explanatory diagram of the operation of a signal processing device (estimation device) according to a comparative example in a learning mode. [Figure 20] FIG. 11 is a block diagram showing the overall configuration of a signal processing system for realizing a change of application of a signal processing device (estimation device) according to a modified example of the first embodiment. [Figure 21] FIG. 11 is an explanatory diagram of a learning mode of the signal processing device (estimation device) according to the first embodiment whose use has been changed. [Figure 22A] FIG. 1 is an explanatory diagram (1) of an inappropriate example (an example in which a signal to be taken (a signal to be extracted) cannot be neatly separated from an input physical quantity) that occurs in a signal processing device (estimation device) according to a comparative example. [Figure 22B] FIG. 11 is an explanatory diagram (2) of an inappropriate example (an example in which a signal to be taken (a signal to be extracted) cannot be neatly separated from an input physical quantity) that occurs in a signal processing device (estimation device) according to a comparative example. [Figure 22C] FIG. 11 is an explanatory diagram (3) of an inappropriate example (an example in which a signal to be taken (a signal to be extracted) cannot be neatly separated from an input physical quantity) that occurs in a signal processing device (estimation device) according to a comparative example. [Figure 22D] FIG. 4 is an explanatory diagram (4) of an inappropriate example (an example in which a signal to be taken (a signal to be extracted) cannot be neatly separated from an input physical quantity) that occurs in a signal processing device (estimation device) according to a comparative example. [Figure 23A] FIG. 1 is an explanatory diagram (1) of a preferred example (an example of a case where a signal to be taken (a signal to be extracted) from an input physical quantity can be neatly separated) realized in the signal processing device (estimation device) according to the first embodiment. [Figure 23B] FIG. 23B is an explanatory diagram (2) of a noise-reduced input physical quantity obtained by reducing noise from the spectrogram of the signal shown in FIG. 23A using the processing mask realized in the signal processing device (estimation device) according to the first embodiment. [Figure 23C] FIG. 3 is an explanatory diagram (3) of a preferred example (an example of a case where a signal to be taken (a signal to be extracted) can be neatly separated from an input physical quantity) realized in the signal processing device (estimation device) according to the first embodiment. [Figure 23D] FIG. 4 is an explanatory diagram (4) of a preferred example (an example of a case where a signal to be taken (a signal to be extracted) can be neatly separated from an input physical quantity) realized in the signal processing device (estimation device) according to the first embodiment. [Figure 23E] FIG. 5 is an illustration of missed signals in a preferred embodiment; [Figure 24] FIG. 13 is a schematic diagram illustrating a method for generating a processing mask. [Diagram 25] FIG. 11 is an explanatory diagram of a method for generating a processing mask. [Figure 26] 11 is an explanatory diagram of a method for generating a noise-reduced input physical quantity. FIG. [Figure 27] 11 is an explanatory diagram of a method for generating a noise-reduced input physical quantity. FIG. [Figure 28] FIG. 2 is an explanatory diagram of a shuffle process in a learning mode of the signal processing device (estimation device) according to the first embodiment. [Figure 29] FIG. 11 is an explanatory diagram of a modified example of the operation of the signal processing device (estimation device) according to the first embodiment in a learning mode. [Diagram 30] FIG. 11 is a block diagram showing the configuration of a signal processing device (estimation device) according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] An embodiment of the present invention (hereinafter, referred to as "the present embodiment") will be described in detail with reference to the drawings. Note that each drawing is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to only the illustrated examples. In addition, in each drawing, common or similar components are given the same reference numerals, and duplicated explanations thereof will be omitted.

[0015] The conventional techniques described in Patent Documents 2 and 3 do not evaluate whether the noise components after separation contain components related to the teacher signal. Such conventional techniques cannot satisfactorily separate the input physical quantity into the noise components and the extracted signal. Therefore, the present invention also aims to provide a signal processing device that evaluates whether the noise components after separation contain components related to the teacher signal and performs learning to satisfactorily separate the input physical quantity into the noise components and the extracted signal.

[0016] Furthermore, the conventional techniques described in Patent Documents 2 and 3 are configured to separate an input physical quantity into a noise component and an extracted signal without considering a phase component related to the input physical quantity. Therefore, even if the conventional techniques separate an input physical quantity into a noise component and an extracted signal, the extracted signal is mixed into the noise component, and the extracted signal cannot be accurately separated. Therefore, the present invention also intends to provide a signal processing device that performs learning to satisfactorily separate an input physical quantity into a noise component and an extracted signal by realizing a configuration that considers the phase component.

[0017] Furthermore, in the conventional techniques of Patent Documents 2 and 3, a transfer function is used to estimate a desired physical quantity, but in the present invention, it does not matter whether a transfer function is used or not.

[0018] [First embodiment] The first embodiment provides a signal processing device including: a noise-reduced input physical quantity generation unit that generates a noise-reduced input physical quantity in which noise components contained in an input physical quantity have been reduced; a selection unit that selects either the input physical quantity or the noise-reduced input physical quantity, or both; and a learning unit that learns weights of a neural network that generates a noise removal mask for removing the noise components from the input physical quantity and the noise-reduced input physical quantity selected by the selection unit.

[0019] <Overall configuration of a signal processing system including a signal processing device (estimation device)> The overall configuration of a signal processing system 100 including a signal processing device 10 (estimation device) according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the signal processing system 100 including the signal processing device 10.

[0020] The signal processing device 10 is a device that performs various processes on a signal. In this embodiment, the signal processing device 10 is described as functioning as an estimation device that estimates an estimated signal of the same dimension (unit) as a teacher signal by multiplying an extracted signal from which noise components have been removed by a transfer function. In addition, in this embodiment, the signal processing device 10 is described as being used for verifying whether or not knocking occurs in an internal combustion engine such as a gasoline engine. However, the signal processing device 10 is not limited to such an application, and can be used for various applications. For example, as described later, the signal processing device 10 can be used for verifying what kind of operating sound is generated by an electronic component to be verified, with the electronic component being one of multiple electronic components mounted on an electronic device as the verification target.

[0021] As shown in FIG. 1, the signal processing system 100 determines whether or not the engine 1 under test is knocking, and includes a sound pressure sensor 4, an in-cylinder pressure sensor 5, a data collection device 6, a monitor 7, headphones 8, a level designation unit 9, and a signal processing device 10.

[0022] As shown in FIG. 1, the engine 1 to be tested is mounted on a vehicle 3. The engine 1 to be tested may be used in a state where it is not mounted on the vehicle 3, for example, in a standalone state. An engine ECU (Electronic Control Unit) 2 that controls the operation of the engine 1 is connected to the engine 1. The engine ECU 2 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random access memory), and other storage devices. The engine ECU 2 controls the operation of the engine 1 while acquiring various information required for the operation control of the engine 1 from outside the engine ECU 2 by processing programs stored in the ROM and storage devices using the CPU.

[0023] The engine ECU 2 outputs angle information representing the current rotation angle of the engine 1 to the data collecting device 6. The angle information includes, for example, a reference angle pulse and a crank angle pulse. The reference angle pulse is a reference pulse output at a reference angle position during one rotation of the crankshaft, and for example, one pulse is output every time the crankshaft rotates once. The crank angle pulse is a pulse output every time the crankshaft rotates at a fixed angle, and for example, when one pulse is output every 1°, 720 pulses are output during two rotations of the crankshaft in one cycle in a four-stroke engine in which four strokes of intake, compression, combustion, and exhaust are one cycle. Note that, if the angle information is input to the data collecting device 6, angle information from an origin sensor that detects the origin of the rotation angle of the crankshaft or an angle sensor that detects the rotation angle of the crankshaft may be input to the data collecting device 6 without going through the engine ECU 2.

[0024] A sound pressure sensor 4 is installed near the engine 1. The sound pressure sensor 4 detects sound generated from the engine 1 and outputs a sound pressure signal based on the detected sound to the data collection device 6. In detail, the sound pressure sensor 4 detects sound pressure, which is an example of a physical quantity based on in-cylinder pressure fluctuations generated in the engine 1, and generates a sound pressure signal indicating the magnitude of the detected sound pressure. Therefore, when the engine 1 is not knocking, the sound pressure signal output from the sound pressure sensor 4 does not include a sound correlated with knocking. On the other hand, when the engine 1 is knocking, the sound pressure signal output from the sound pressure sensor 4 includes a sound correlated with knocking.

[0025] An in-cylinder pressure sensor 5 that detects the in-cylinder pressure of the engine 1 is attached to the engine 1. The in-cylinder pressure sensor 5 outputs an in-cylinder pressure signal including a waveform component corresponding to the combustion gas vibration in the cylinder to a data collection device 6. Here, when the engine 1 is not knocking, the in-cylinder pressure signal output from the in-cylinder pressure sensor 5 does not include a component correlated with the knocking. On the other hand, when the engine 1 is knocking, the in-cylinder pressure signal output from the in-cylinder pressure sensor 5 includes a component correlated with the knocking. The in-cylinder pressure sensor 5 may be integrated with the ignition plug, or may be configured separately from the ignition plug.

[0026] The data collecting device 6 receives the sound pressure signal from the sound pressure sensor 4 and performs A / D conversion. The data collecting device 6 also obtains current angle information from the engine ECU 2 at the timing of receiving the sound pressure signal. The data collecting device 6 then obtains sound pressure signals for one cycle of the engine 1 based on the angle information. Thus, in the case of a four-stroke single cylinder engine, the data collecting device 6 generates sound pressure signals in a number according to the rotation speed of the engine 1 per unit time, for example, 1500 sound pressure signals per minute if the rotation speed is 3000 r / min. The data collecting device 6 also associates angle information with some or all of the sound pressure signals. The data collecting device 6 then outputs the sound pressure signals associated with the angle information to the signal processing device 10. The data collecting device 6 may temporarily hold or temporarily store the sound pressure signals before outputting them to the signal processing device 10. The data collecting device 6 may also associate time information with the sound pressure signals.

[0027] The monitor 7 displays the knocking sound of the engine 1 extracted by the signal processing device 10, the estimated in-cylinder pressure of the engine 1, and the result of the determination of the presence or absence of knocking by the signal processing device 10. An example of the monitor 7 is a general flat panel display.

[0028] The headphones 8 are a sound emitting unit that emits sound. The headphones 8 are worn on the head of an inspector when inspecting the condition of a test object (in this embodiment, the engine 1) in a sensory test mode described later.

[0029] The level designation unit 9 is an input unit that receives level designation information that designates an increase or decrease level for a signal. The level designation unit 9 is configured with, for example, a touch panel display, a numeric keypad, a dedicated switch, etc. In this embodiment, when the level of an extraction signal (extracted knocking sound 91a (FIG. 11B)) is changed to generate a level-changed extraction signal (level-changed knocking sound 91aa (FIG. 12A)), the level designation unit 9 is operated by people around the inspector, including the inspector, to designate (input) the increase or decrease level of the extraction signal.

[0030] The signal processing device 10 learns the weights W (FIG. 3B) and transfer function H (FIG. 3A) of a neural network 94 (FIG. 3A) that generates a noise removal mask α. Hereinafter, the "neural network 94 that generates the noise removal mask α" may be referred to as the "mask generation network 94A (FIG. 3A)". Also, the "weights W of the mask generation network 94A" may be referred to as the "weights W of the neural network". Herein, the noise removal mask α is a real or complex matrix for removing noise components from an input physical quantity that contains the noise components. The noise removal mask α changes according to the input physical quantity. Also, the transfer function H is a complex weight vector, and is interpreted as the inverse of the structural damping correction amount of the engine 1. The structural damping correction amount is a transfer characteristic in which vibrations caused by the in-cylinder pressure during engine combustion pass through the engine 1, become sound, and reach the sound pressure sensor. The combustion noise level of the engine 1 is obtained by multiplying the frequency component of the in-cylinder pressure of the engine 1 by the structural damping correction amount.

[0031] Therefore, the in-cylinder pressure of the engine 1 can be obtained by multiplying the combustion noise contained in the nearby sound of the engine 1, which is the sound pressure, by the inverse of the structural damping correction amount. Therefore, the in-cylinder pressure of the engine 1 can be estimated from the nearby sound of the engine 1.

[0032] The principle of estimating the in-cylinder pressure of the engine 1 from the nearby sound of the engine 1 is described, for example, in the above-mentioned Patent Document 2. According to the above-mentioned Patent Document 2, "First, the nearby sound y of the engine 1 and the actually measured in-cylinder pressure x of the engine 1 are collected as teacher data (teacher signal). (Omitted) Using the nearby sound y of the engine 1 and the actually measured in-cylinder pressure x of the engine 1, the weights and transfer function of the neural network that generates a mask, which is an unknown quantity, are learned by the neural network. Then, using the mask and transfer function generated by the learned neural network, the in-cylinder pressure x of the engine 1 is estimated from the nearby sound y of the engine 1 measured during testing."

[0033] The signal processing device 10 extracts a knocking sound from sounds in the vicinity of the engine 1 using the noise removal mask α and transfer function H generated by the trained neural network 94 (FIG. 3A), estimates the knocking in-cylinder pressure of the engine 1, and determines the presence or absence of knocking based on the extracted knocking sound. Then, the signal processing device 10 displays the extracted knocking sound, the estimated knocking in-cylinder pressure of the engine 1, and the determination result of the presence or absence of knocking on the monitor 7.

[0034] Here, the signal processing device 10 is composed of a CPU, a ROM, a RAM, other storage devices, etc. The signal processing device 10 performs arithmetic processing of programs stored in the ROM and storage devices using the CPU. Note that the signal processing device 10 may be a personal computer (PC) or the like having a program that executes the following processes.

[0035] In this embodiment, the signal processing device 10 operates in five operation modes, namely, a learning mode, a threshold calculation mode, a judgment mode, a separation mode, and a sensory test mode. The first learning mode is an operation mode for learning the weight W (FIG. 3B) and the transfer function H (FIG. 3A) of the neural network that generates the noise removal mask α (FIG. 3A). The second threshold calculation mode is an operation mode for calculating a threshold for judging the presence or absence of knocking after learning the weight W and the transfer function H of the neural network that generates the noise removal mask α. The third judgment mode is an operation mode for extracting an extraction signal (in this embodiment, an extracted knocking sound 91a) from an input physical quantity (in this embodiment, an engine vicinity sound 90) using a noise removal mask α generated by inputting an input physical quantity to the trained neural network 94 (FIG. 3A), and judging the presence or absence of knocking. The fourth separation mode is an operation mode for separating the input physical quantity into an extraction signal and a noise component (in this embodiment, noise 91b). The fifth sensory test mode is an operation mode in which an inspector listens to the processed sound described below and determines data to be used in a threshold calculation mode for a target sound to be inspected (knocking sound in this embodiment) based on the inspector's sense of hearing. In the fifth sensory test mode, a threshold is calculated from a sound that is outside the allowable range, for example.

[0036] These five operation modes can be arbitrarily switched. For example, the operation modes of the signal processing device 10 can be switched by a management device (not shown) via a CAN (Controller Area Network). The operation modes of the signal processing device 10 may also be switched by using operation means such as a mouse and a keyboard (not shown).

[0037] In the learning mode, the engine 1 is operated under operating conditions where knocking occurs and operating conditions where knocking does not occur, and the data collection device 6 collects the in-cylinder pressure signal as teacher data (teacher signal). At this time, the data collection device 6 receives the in-cylinder pressure signal from the in-cylinder pressure sensor 5, A / D converts it, and associates it with the sound pressure signal.

[0038] In the case of the threshold calculation mode, the engine 1 is operated under operating conditions in which knocking does not occur, and the data collecting device 6 collects sound pressure signals.

[0039] In the determination mode, a knocking sound 91a (extracted signal) is extracted from the engine vicinity sound 90 (input physical quantity) using a noise removal mask α generated by the trained neural network, and the signal processing device 10 determines whether or not knocking occurs based on a threshold value.

[0040] In addition, in the case of the threshold calculation mode or the judgment mode, it is not necessary for the data collection device 6 to collect the in-cylinder pressure signal. That is, in the case of the learning mode, both the sound pressure sensor 4 and the in-cylinder pressure sensor 5 operate, but in the case of the threshold calculation mode or the judgment mode, only the sound pressure sensor 4 operates.

[0041] In the separation mode, the engine 1 is operated, and the signal processing device 10 inputs the engine vicinity sound 90 (input physical quantity) to the learned neural network to generate a noise removal mask α, and separates the noise 91b (noise component) and the extracted knocking sound 91a (extracted signal). At this time, the signal processing device 10 stores the noise 91b in the noise component storage unit 26c and the extracted knocking sound 91a in the extracted signal storage unit 26d. In the present invention, the separation performance is improved by learning the weight W and transfer function H of the neural network that generates the noise removal mask α, taking into account the amplitude and phase related to the input physical quantity. The noise removal mask α is implemented as a real number or a complex number, and the transfer function H is implemented as a complex number. This allows the signal processing device 10 to perform learning to satisfactorily separate the input physical quantity into the noise component and the extracted signal.

[0042] In the sensory test mode, the signal processing device 10 receives level designation information from the level designation unit 9, and generates a processed sound using the noise 91b (noise component) and the extracted knocking sound 91a (extracted signal) based on the level designation information.

[0043] Even if the amplitude spectrum is the same, the signal waveform changes significantly depending on the phase spectrum, which has a significant impact on the auditory impression. Therefore, in order to generate a processed sound with good quality, it is important to acquire an extracted signal (in this embodiment, the extracted knocking sound 91a) used for the processed sound, taking into account the phase. In other words, the signal processing device 10 improves separation performance by learning the weight W and transfer function H of the neural network that generates the noise removal mask α, taking into account the amplitude and phase related to the input physical quantity, and can generate a processed sound with good quality (a processed sound in which the knocking sound is natural to the auditory sense).

[0044] <Configuration of signal processing device (estimation device)> The configuration of the signal processing device 10 (estimation device) will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the signal processing device 10. As shown in Fig. 2, the signal processing device 10 includes a signal extraction unit 11, a spectrogram calculation unit 12, a signal storage unit 13, a switch 14, a learning processing unit 20 (learning device), a judgment processing unit 30, a separation unit 40, and a signal synthesis unit 50. Here, a sound pressure signal and angle information associated with this sound pressure signal are input from a data collection device 6 to the signal processing device 10. Furthermore, in the case of a learning mode, an in-cylinder pressure signal is input from the data collection device 6 to the signal processing device 10.

[0045] The signal cutting unit 11 cuts out a sound pressure signal in a predetermined cutting angle range from the input sound pressure signal based on the angle information input from the data collecting device 6. For example, the cutting angle range is an angle range of about -10 to 90° of ATDC (After Top Dead Center). In this embodiment, since the cutting is performed based on TDC (Top Dead Center), the cutting angle range remains fixed even if the ignition timing is changed. However, the cutting angle range may be changed according to the change in the ignition timing. When the signal cutting unit 11 cuts out the sound pressure signal, it outputs the cut sound pressure signal to the spectrogram calculating unit 12.

[0046] The spectrogram calculation unit 12 calculates a spectrogram of the sound pressure signal by performing a Short Time Fourier Transform (STFT) on the sound pressure signal extracted by the signal extraction unit 11. The Short Time Fourier Transform is performed by, for example, a Fast Fourier Transform (FFT) that quickly calculates a Discrete Fourier Transform. Thereafter, the spectrogram calculation unit 12 writes the spectrogram of the sound pressure signal into the signal storage unit 13.

[0047] The signal storage unit 13 is a storage device such as a memory, HDD (Hard Disk Drive), or SSD (Solid State Drive) that stores a spectrogram of the sound pressure signal converted by the spectrogram calculation unit 12. In the learning mode, the signal storage unit 13 stores the in-cylinder pressure signal (observed knocking in-cylinder pressure 93 (FIG. 3A)) input from the data collection device 6 as teacher data (teacher signal). Here, the "knocking in-cylinder pressure" refers to the knocking component superimposed on the in-cylinder pressure signal. The knocking in-cylinder pressure is obtained by subjecting the in-cylinder pressure to a high-pass filter process that passes a frequency band equal to or higher than the frequency component of knocking.

[0048] The switch 14 can be arbitrarily switched to a learning mode connection part M1, a threshold calculation mode connection part M2, a determination mode connection part M3, a separation mode connection part M4, or a sensory test mode connection part M5, which correspond to the five operation modes described above. The signal processing device 10 can output a signal stored in the signal storage part 13 to an arbitrary output destination by switching the switch 14 according to the operation mode of the signal processing device 10.

[0049] For example, in the learning mode, the signal processing device 10 connects the switch 14 to a connection section M1 for the learning mode, and outputs the spectrogram of the sound pressure signal and the in-cylinder pressure signal stored in the signal storage section 13 to a noise reduction input physical quantity generating section 15 and a selection section 16, which will be described later. In the threshold calculation mode, the signal processing device 10 connects the switch 14 to a connection section M2 for the threshold calculation mode, and outputs the spectrogram of the sound pressure signal stored in the signal storage section 13 to a first estimation section 23, which will be described later. In the determination mode, the signal processing device 10 connects the switch 14 to a connection section M3 for the determination mode, and outputs the spectrogram of the sound pressure signal stored in the signal storage section 13 to a second estimation section 31 (extracted signal estimation section), which will be described later. In the separation mode, the signal processing device 10 connects the switch 14 to a connection section M4 for the separation mode, and outputs the spectrogram of the sound pressure signal stored in the signal storage section 13 to a separation section 40, which will be described later. In addition, in the case of the sensory test mode, the signal processing device 10 connects the switch 14 to the sensory test mode connection section M5 and outputs an execution instruction signal for the sensory test mode, which is generated in advance by a signal generation section (not shown) and stored in the signal storage section 13, to a signal adjustment section 51 (described later). Here, the execution instruction signal for the sensory test mode is described as being generated in advance by a signal generation section (not shown) and stored in the signal storage section 13. However, the execution instruction signal for the sensory test mode may be generated by a signal generation section (not shown) when the switch 14 is connected to the sensory test mode connection section M5, and output from the signal generation section (not shown) to the signal adjustment section 51 (described later) without passing through the signal storage section 13.

[0050] <Learning processing unit> In the learning mode, the learning processing unit 20 performs a learning process (FIG. 14A) for learning a weight W and a transfer function H of a neural network 94 (FIG. 3A) that generates a noise removal mask α, and in the threshold calculation mode, performs a threshold calculation process (FIG. 15A) for calculating a threshold used for threshold determination of an extracted knocking sound of the engine 1. As shown in FIG. 2, the learning processing unit 20 includes a noise reduction input physical quantity generating unit 15, a selecting unit 16, a learning unit 21, a learned parameter storage unit 22, a first estimating unit 23, a threshold calculating unit 24, a threshold storage unit 25, a teacher signal storage unit 26a, an estimated signal storage unit 26b, a noise component storage unit 26c, and an extracted signal storage unit 26d.

[0051] The noise-reduction input physical quantity generation unit 15 generates a noise-reduction input physical quantity 90A (FIG. 3A) by reducing a noise component (in this embodiment, noise 91b (FIG. 3A)) included in an input physical quantity (in this embodiment, near-engine sound 90 (FIG. 3A)). The noise-reduction input physical quantity generation unit 15 has a function of generating a processing mask β for generating the noise-reduction input physical quantity 90A from the input physical quantity (in this embodiment, near-engine sound 90 (FIG. 3A)). The processing mask β is obtained by filtering the noise removal mask α. The processing mask β will be described later.

[0052] The selection unit 16 has a shuffle function that supplies a data set in which the input physical quantity (near-engine sound 90 (FIG. 3A)) and the noise-reduction input physical quantity 90A (FIG. 3A) are shuffled to the learning unit 21. Note that "shuffle" here means to mix the order of the physical quantities in a random order. The selection unit 16 selects either the input physical quantity or the noise-reduction input physical quantity, or both, and supplies the selected data to the learning unit 21.

[0053] In the learning mode, the learning unit 21 learns the weights of the neural network 94 that generates a noise removal mask α for removing a noise component (noise 91b (FIG. 3A)) from the input physical quantity (engine vicinity sound 90 (FIG. 3A)) and the noise reduction input physical quantity 90A (FIG. 3A) selected by the selection unit 16. The learning unit 21 also learns a transfer function H (FIG. 3A) that converts an extracted signal (extracted knocking sound 91a (FIG. 3A)) extracted by the generated noise removal mask α to an estimated knocking in-cylinder pressure 92 of the engine 1 when knocking occurs. In this embodiment, the learning unit 21 learns the weights W and transfer function H of the neural network 94 that generates the noise removal mask α using the spectrogram of the sound pressure signal and the in-cylinder pressure signal input from the signal storage unit 13. Thereafter, the learning unit 21 writes the weights W and transfer function H of the neural network 94 that generates the learned noise removal mask α to the learned parameter storage unit 22.

[0054] The learned parameter storage unit 22 is a storage device such as a memory, HDD, or SSD that stores learned parameters (weights W and transfer function H of the neural network that generates the noise removal mask α).

[0055] In the threshold calculation mode, the first estimation unit 23 extracts the extracted knocking sound 91a of the engine 1 from the spectrogram of the sound pressure signal using the noise removal mask α generated by the neural network 94. The extracted knocking sound 91a extracted by the first estimation unit 23 is used when calculating a threshold value described later. Specifically, the first estimation unit 23 inputs the spectrogram of the sound pressure signal input from the signal storage unit 13 to the neural network 94 reflecting the weight W of the neural network that generates the noise removal mask α of the learned parameter storage unit 22. Then, the noise removal mask α generated by the learned neural network extracts the extracted knocking sound 91a from the spectrogram of the sound pressure signal. After that, the first estimation unit 23 outputs the extracted knocking sound 91a to the threshold calculation unit 24.

[0056] The threshold calculation unit 24 calculates a threshold based on the extracted knocking sound 91a of the engine 1 extracted by the first estimation unit 23. Specifically, the threshold calculation unit 24 sums up the absolute values ​​of the spectrogram of the extracted knocking sound 91a for each predetermined time (for example, one cycle of the engine 1). For example, in the engine 1 having a plurality of cylinders, when the extracted knocking sounds 91a are summed up, one score is obtained for each cylinder. Next, the threshold calculation unit 24 calculates the median of the extracted knocking sounds 91a summed up for each predetermined time. For example, the threshold calculation unit 24 calculates the median of the sum of the extracted knocking sounds 91a for all cycles. At this time, the threshold calculation unit 24 adds a preset margin of any value to the median to obtain the threshold. The threshold calculation unit 24 may sum up the extracted knocking sounds 91a for each cylinder to obtain the median, and calculate the threshold for each cylinder. The threshold calculation unit 24 may also sum up the extracted knocking sounds 91a for each cylinder, find the median value for all cylinders, and calculate a common threshold for all cylinders. Alternatively, the sum of the extracted knocking sounds 91a for the processed sounds 91c that are determined to be unacceptable by an inspector in a sensory test described later may be calculated, and any value equal to or less than the sum may be set as the threshold. Thereafter, the threshold calculation unit 24 writes the calculated threshold into the threshold storage unit 25.

[0057] The threshold storage unit 25 is a storage device such as a memory, HDD, or SSD that stores the threshold calculated by the threshold calculation unit 24.

[0058] The teacher signal storage unit 26a is a storage device such as a memory, HDD, or SSD that stores a teacher signal (in this embodiment, the observed knocking in-cylinder pressure 93 (FIG. 3A)).

[0059] The estimated signal storage unit 26b is a storage device such as a memory, HDD, or SSD that stores an estimated signal (in this embodiment, an estimated knocking in-cylinder pressure 92 (FIG. 3A)).

[0060] The noise component storage unit 26c is a storage device such as a memory, HDD, or SSD that stores noise components (in this embodiment, noise 91b (FIG. 6A)) included in the input physical quantity.

[0061] The extracted signal storage unit 26d is a storage device such as a memory, HDD, or SSD that stores an extracted signal (in this embodiment, an extracted knocking sound 91a (FIG. 6A)) obtained by removing the noise component from the input physical quantity including the noise component.

[0062] <Determination processing unit> In the determination mode, the determination processing unit 30 extracts the knocking sound of the engine 1 from the spectrogram of the sound pressure signal and performs a determination process (FIG. 16) to determine whether knocking occurs. As shown in FIG. 2, the determination processing unit 30 includes a second estimation unit 31 (extracted signal estimation unit) and a threshold determination unit 32.

[0063] In the determination mode, the second estimation unit 31 (extracted signal estimation unit) extracts an extracted knocking sound 91a (extracted signal) from the spectrogram of the sound pressure signal using a noise removal mask α generated by the neural network 94. The extracted knocking sound 91a extracted by the second estimation unit 31 is used for threshold determination, which will be described later. Note that the processing content of the second estimation unit 31 is similar to that of the first estimation unit 23, and therefore a description thereof will be omitted. Thereafter, the second estimation unit 31 outputs the extracted knocking sound 91 a to the threshold determination unit 32 .

[0064] In the determination mode, the threshold determination unit 32 determines whether or not knocking occurs by threshold determination between the threshold stored in the threshold storage unit 25 and the extracted knocking sound 91a extracted by the second estimation unit 31. Specifically, the threshold determination unit 32 sums up the absolute values ​​of the spectrogram of the extracted knocking sound 91a at predetermined time intervals, similar to the threshold calculation unit 24. The threshold determination unit 32 then compares the summed extracted knocking sound 91a with the threshold, and determines that knocking occurs if the summed extracted knocking sound 91a exceeds the threshold, and determines that knocking does not occur if the summed extracted knocking sound 91a is equal to or smaller than the threshold. Thereafter, the threshold determination unit 32 outputs the determination result of whether or not knocking occurs and the extracted knocking sound 91a input from the second estimation unit 31 to the monitor 7 (FIG. 1).

[0065] <Separation part> The separation unit 40 separates the input physical quantity into a noise component and an extracted signal by using the noise removal mask α. In this embodiment, as shown in Fig. 6A, the separation unit 40 separates a near-engine sound 90 as an input physical quantity into noise 91b, which is a noise component, and an extracted knocking sound 91a, which is an extracted signal.

[0066] <Signal synthesis section> The signal synthesis unit 50 synthesizes various signals to generate a processed sound. In this embodiment, as shown in Figs. 11A to 12C, the signal synthesis unit 50 changes the level of the extracted knocking sound 91 (extracted signal) and synthesizes it with noise 91b (noise components) separated from the engine vicinity sound 90 (input physical quantity) to generate a processed sound. The signal synthesis unit 50 has a signal adjustment unit 51 and a signal output unit 52. The signal adjustment unit 51 changes the level of the extracted knocking sound 91 according to the rising or falling level of the signal designated (input) by the level designation unit 9, and synthesizes the extracted knocking sound 91 (level-changed extracted signal) whose level has been changed with the noise components to generate a processed sound. The signal output unit 52 outputs the processed sound (signal) to the sound emission unit (headphones 8) to cause the sound emission unit to emit the processed sound.

[0067] <Learning mode operation> The operation of the signal processing device 10 in the learning mode will be described with reference to Figs. 3A, 3B, 8A, 8B, 8C, and 9. Fig. 3A is an explanatory diagram of the learning mode. Fig. 3B is an explanatory diagram of the operation of the signal processing device 10 in the learning mode. Fig. 8A is an explanatory diagram showing the relationship between a noise component and a teacher signal during learning. Fig. 8B is an explanatory diagram showing the relationship between an extracted signal and a teacher signal during learning. Fig. 8C is an explanatory diagram showing the relationship between an extracted signal and an estimated signal during learning. Fig. 9 is an explanatory diagram of learning the weight W of the neural network that generates the noise removal mask α in the first embodiment.

[0068] 3A, in the learning mode, in the signal processing device 10, a near-engine sound 90 (input physical quantity) is supplied to the noise-reduction input physical quantity generation unit 15 and the selection unit 16. In addition, as shown in FIG 3B, an observed knocking in-cylinder pressure 93 (teacher signal) is supplied to the noise-reduction input physical quantity generation unit 15 and the selection unit 16.

[0069] The noise reduction input physical quantity generation unit 15 generates a noise reduction input physical quantity 90A by reducing noise 91b contained in the near-engine sound 90 from the near-engine sound 90 using a processing mask β obtained by filtering the noise reduction mask α, and supplies the noise reduction input physical quantity 90A to the selection unit 16. The noise reduction mask α is generated using the weighting W of the neural network every time either one or both of the near-engine sound 90 (input physical quantity) and the noise reduction input physical quantity 90A are input to the learning unit 21 when the learning unit 21 learns the weighting W of the neural network.

[0070] The selection unit 16 selects either or both of the near-engine sound 90 and the noise-reduction input physical quantity 90A generated by the noise-reduction input physical quantity generation unit 15, and supplies them to the learning unit .

[0071] The learning unit 21 learns the weight W of the neural network that generates a noise removal mask α for removing noise 91b from the sound near the engine 90 and the noise reduction input physical quantity 90A selected by the selection unit 16. At this time, the learning unit 21 separates the sound near the engine 90 and the noise reduction input physical quantity 90A selected by the selection unit 16 into noise 91b and an extracted knocking sound 91a. At this time, the learning unit 21 multiplies the sound near the engine 90 and the noise reduction input physical quantity 90A selected by the selection unit 16 by the noise removal mask α to obtain the extracted knocking sound 91a. In addition, the learning unit 21 obtains the noise 91b by subtracting the extracted knocking sound 91a from the sound near the engine 90 and the noise reduction input physical quantity 90A selected by the selection unit 16. In this embodiment, the noise removal mask α represents the proportion of the knocking sound included in the sound near the engine 90 and the noise reduction input physical quantity 90A selected by the selection unit 16 and the phase component (phase correction amount). The phase component will be described later with reference to FIGS. 10A and 10B.

[0072] When learning the weight W of the neural network, the learning unit 21 acquires a noise elimination mask α and supplies the noise elimination mask α to the noise reduction input physical quantity generation unit 15. The noise reduction input physical quantity generation unit 15 performs a filter process on the noise elimination mask α to acquire a processing mask β. Then, the noise reduction input physical quantity generation unit 15 generates a noise reduction input physical quantity 90A from the near-engine sound 90 using the processing mask β, and supplies the noise reduction input physical quantity 90A to the selection unit 16.

[0073] The learning unit 21 learns the weight W of the neural network, and also learns a transfer function H for converting the extracted knocking sound 91a (extracted signal) into an estimated knocking in-cylinder pressure 92 (estimated signal) of the same dimension (unit) as the observed knocking in-cylinder pressure 93 (teacher signal) while taking into account the phase. In other words, when learning the weight W of the neural network and the transfer function H, the learning unit 21 learns the noise removal mask α and the transfer function H while taking into account the amplitude and phase components related to the near-engine sound 90 (input physical quantity). In this embodiment, the learning unit 21 performs an inverse short-time Fourier transform (ISTFT) and a fast Fourier transform (FFT) on the extracted knocking sound 91a, multiplies it by the transfer function H, and performs an inverse fast Fourier transform (IFFT) and a short-time Fourier transform (STFT) to convert the extracted knocking sound 91a into the estimated knocking in-cylinder pressure 92. In this embodiment, the transfer function H is an amplitude (gain) and a phase component for converting the extracted knocking sound 91a into an estimated knocking in-cylinder pressure 92. The phase component will be described later with reference to Figs. 10A and 10B.

[0074] In this embodiment, as shown in FIG. 3A, the learning unit 21 learns the weights of the neural network so that the correlation between the noise 91b (noise component) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced and the correlation between the extracted knocking sound 91a (extracted signal) and the teacher signal (observed knocking in-cylinder pressure 93) is increased. Specifically, the learning unit 21 learns the weights W of the neural network that generates the noise removal mask α so that the coherence between the first signal Sfi acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise 91b (noise component) and the observed knocking in-cylinder pressure 93 (teacher signal) is reduced and the coherence between the second signal Sse acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted knocking sound 91a (extracted signal) and the observed knocking in-cylinder pressure 93 (teacher signal) is increased. This allows the signal processing device 10 to remove the sound caused by the in-cylinder pressure from the noise 91b.

[0075] Fig. 3B shows the operation of the signal processing device 10 in the learning mode. The bold frame in Fig. 3B indicates the components that operate in the learning mode. The bold arrow in Fig. 3B indicates the signals that are output in the learning mode.

[0076] As shown in FIG. 3B , in the learning mode, the signal processing device 10 connects the switch 14 to the learning mode connection unit M1 and outputs the spectrogram of the near-engine sound 90 and the observed knocking in-cylinder pressure 93 stored in the signal storage unit 13 to the noise reduction input physical quantity generation unit 15 and the selection unit 16.

[0077] In response to this, the noise reduction input physical quantity generation unit 15 uses the processing mask β to generate a noise reduction input physical quantity 90A from the engine vicinity sound 90 and supplies it to the selection unit 16. The selection unit 16 selects either or both of the engine vicinity sound 90 and the noise reduction input physical quantity 90A generated by the noise reduction input physical quantity generation unit 15, and supplies them to the learning unit 21. The learning unit 21 learns the weighting W of the neural network using the engine vicinity sound 90 and the noise reduction input physical quantity 90A selected by the selection unit 16. In addition, the learning unit 21 learns a transfer function H that converts the extracted knocking sound 91a extracted with the generated noise removal mask α to an estimated knocking in-cylinder pressure 92 of the engine 1 at the time of knocking occurrence. At this time, every time a new engine vicinity sound 90 and a noise reduction input physical quantity 90A selected by the selection unit 16 are input, the learning unit 21 generates a noise removal mask α corresponding to the one selected by the selection unit 16.

[0078] Then, the learning unit 21 stores the learned parameters (weight W and transfer function H of the neural network that generates the noise removal mask α) in the learned parameter storage unit 22. The learning unit 21 also stores the observed knocking in-cylinder pressure 93 in the teacher signal storage unit 26a, and stores the estimated knocking in-cylinder pressure 92 in the estimated signal storage unit 26b.

[0079] <Operation in threshold calculation mode> The operation of the signal processing device 10 in the threshold calculation mode will be described with reference to Figures 4A and 4B. Figure 4A is an explanatory diagram of the threshold calculation mode. Figure 4B is an explanatory diagram of the operation of the signal processing device 10 in the threshold calculation mode.

[0080] As shown in Fig. 4B, in the threshold calculation mode, unlike the learning mode in Fig. 3B, the learning unit 21 of the signal processing device 10 is in a stopped state. Instead, as shown in Fig. 4B, the first estimation unit 23 and the threshold calculation unit 24 of the learning processing unit 20 operate to calculate the threshold T used for threshold determination of the extracted knocking sound 91a of the engine 1. The bold frame in Fig. 4B indicates the components that operate in the threshold calculation mode. Also, the bold arrow in Fig. 4B indicates the signals that are output in the threshold calculation mode.

[0081] As shown in FIG. 4B, in the threshold calculation mode, the signal processing device 10 connects the switch 14 to the threshold calculation mode connection unit M2 and outputs the spectrogram of the near-engine sound 90 stored in the signal storage unit 13 to the first estimation unit 23.

[0082] In response to this, the first estimation unit 23 acquires the weight W of the neural network that generates the noise removal mask α from the learned parameter storage unit 22. Then, the first estimation unit 23 extracts an extracted knocking sound 91a of the engine 1 from the near-engine sound 90 using the noise removal mask α generated by the neural network 94 (FIG. 4A). FIG. 4A shows an overview of the operation of the first estimation unit 23 at this time. Thereafter, the first estimation unit 23 outputs the extracted knocking sound 91a to the threshold calculation unit 24.

[0083] In response to this, the threshold calculation unit 24 sums up the absolute values ​​of the spectrograms of the extracted knocking sound 91a in each cycle of the engine 1, and adds a preset margin to the sum to calculate the threshold T. Then, the threshold calculation unit 24 stores the threshold T in the threshold storage unit 25.

[0084] <Operation in judgment mode> The operation of the signal processing device 10 in the determination mode will be described with reference to Figures 5A and 5B. Figure 5A is an explanatory diagram of the determination mode. Figure 5B is an explanatory diagram of the operation of the signal processing device 10 in the determination mode.

[0085] As shown in Fig. 5B, in the judgment mode, unlike the learning mode in Fig. 3B, the learning unit 21 of the signal processing device 10 is in a stopped state. Instead, as shown in Fig. 5B, the second estimation unit 31 (extracted signal estimation unit) and the threshold judgment unit 32 of the judgment processing unit 30 operate to judge the presence or absence of knocking. The bold frame in Fig. 5B indicates the components that operate in the judgment mode. Also, the bold arrow in Fig. 5B indicates the signals that are output in the judgment mode.

[0086] As shown in FIG. 5B, in the determination mode, the signal processing device 10 connects the switch 14 to the determination mode connection unit M3 and outputs the spectrogram of the near-engine sound 90 stored in the signal storage unit 13 to the second estimation unit 31.

[0087] In response to this, the second estimation unit 31 acquires the weight W of the neural network that generates the noise removal mask α from the learned parameter storage unit 22. Then, the second estimation unit 31 extracts an extracted knocking sound 91a from the near-engine sound 90 using the noise removal mask α generated by the neural network 94 (FIG. 5A). FIG. 5A shows an overview of the operation of the second estimation unit 31 at this time. Thereafter, the second estimation unit 31 outputs the extracted knocking sound 91a to the threshold determination unit 32.

[0088] In response to this, the threshold determination unit 32 obtains the threshold T from the threshold storage unit 25, and determines the presence or absence of knocking by comparing the sum of the absolute values ​​of the extracted knocking sounds 91a with the threshold T. Then, the threshold determination unit 32 outputs, for example, the determination result of the presence or absence of knocking and a waveform diagram showing the relationship between the extracted knocking sound 91a and the threshold T to the monitor 7 for display.

[0089] <Operation in separate mode> The operation of the signal processing device 10 in the separation mode will be described with reference to Figures 6A and 6B. Figure 6A is an explanatory diagram of the separation mode. Figure 6B is an explanatory diagram of the operation of the signal processing device 10 in the separation mode.

[0090] As shown in Fig. 6B, in the separation mode, unlike the learning mode in Fig. 3B, the learning unit 21 of the signal processing device 10 is in a stopped state. Instead, as shown in Fig. 6B, the separation unit 40 operates to separate the near-engine sound 90 into an extracted knocking sound 91a and noise 91b. The bold frame in Fig. 6B indicates the components that operate in the separation mode. Also, the bold arrow in Fig. 6B indicates the signals that are output in the separation mode.

[0091] As shown in FIG. 6B, in the separation mode, the signal processing device 10 connects the switch 14 to the separation mode connection section M4 and outputs the spectrogram of the near-engine sound 90 stored in the signal storage section 13 to the separation section 40.

[0092] In response to this, the separation unit 40 acquires the weight W of the neural network that generates the noise removal mask α from the learned parameter storage unit 22. Then, the separation unit 40 separates the near-engine sound 90 into an extracted knocking sound 91a and noise 91b using the noise removal mask α generated by the neural network 94 (FIG. 6A). FIG. 6A shows an overview of the operation of the separation unit 40 at this time. Thereafter, the separation unit 40 stores the extracted knocking sound 91a (extracted signal) in the extracted signal storage unit 26d, and stores the noise 91b in the noise component storage unit 26c.

[0093] <Operation in sensory test mode> The operation of the signal processing device 10 in the sensory test mode will be described with reference to Fig. 7A and Fig. 7B. Fig. 7A is an explanatory diagram of the sensory test mode. Fig. 7B is an explanatory diagram of the operation of the signal processing device 10 in the sensory test mode. The sensory test mode is a mode for calculating a threshold value T based on the auditory sense of the examiner.

[0094] As shown in Fig. 7B, in the sensory test mode, unlike the learning mode in Fig. 3B, the learning unit 21 of the signal processing device 10 is in a stopped state. Instead, as shown in Fig. 7B, the signal adjustment unit 51 and the signal output unit 52 of the signal synthesis unit 50, and the first estimation unit 23 and the threshold calculation unit 24 of the learning processing unit 20 operate to calculate a threshold T based on the auditory sense of the examiner. The bold frame in Fig. 7B indicates the components that operate in the sensory test mode. Also, the bold arrow in Fig. 7B indicates the signal output in the sensory test mode.

[0095] 7B, in the sensory test mode, the signal processing device 10 connects the switch 14 to the sensory test mode connection section M5, thereby outputting an execution instruction signal for the sensory test mode, which is generated in advance by a signal generation section (not shown) and stored in the signal storage section 13, to the signal adjustment section 51. However, the execution instruction signal for the sensory test mode may be generated by a signal generation section (not shown) when the switch 14 is connected to the sensory test mode connection section M5, and output from the signal generation section (not shown) to the signal adjustment section 51 without passing through the signal storage section 13.

[0096] In response to this, the signal adjustment unit 51 receives level designation information from the level designation unit 9, and acquires the extracted knocking sound 91a stored in the extracted signal storage unit 26d and the noise 91b stored in the noise component storage unit 26c. Then, the signal adjustment unit 51 generates a processed sound 91c using the extracted knocking sound 91a and the noise 91b based on the level designation information. At this time, the signal adjustment unit 51 raises or lowers the level (volume) of the extracted knocking sound 91a by an amount designated by the level designation information, and then combines it with the noise 91b to generate the processed sound 91c and outputs it to the signal output unit 52. The signal output unit 52 outputs the processed sound 91c to the headphones 8 (sound output unit) to emit the sound.

[0097] Further, the signal adjustment unit 51 outputs the processed sound 91c to the first estimation unit 23. The first estimation unit 23 acquires the weight W of the neural network that generates the noise removal mask α from the learned parameter storage unit 22. Then, the first estimation unit 23 extracts the extracted knocking sound 91a from the processed sound 91c using the noise removal mask α generated by the neural network 94 (FIG. 7A). FIG. 7A shows an overview of the operation of the first estimation unit 23 at this time. After that, the first estimation unit 23 outputs the extracted knocking sound 91a to the threshold calculation unit 24. The threshold calculation unit 24 sums up the absolute values ​​of the spectrogram of the extracted knocking sound 91a in each cycle of the engine 1, and sets an arbitrary value equal to or less than the sum value as the threshold T. Then, the threshold calculation unit 24 stores the threshold T in the threshold storage unit 25.

[0098] Incidentally, the conventional techniques described in Patent Documents 2 and 3 do not include a function for measuring the degree of separation of sounds in the objective function during learning of the neural network, so there is a possibility that the knocking sound (target sound) may be mixed into the noise (sounds other than the knocking sound (background sound)) removed from the engine vicinity sound (input physical quantity). In other words, the conventional techniques described in Patent Documents 2 and 3 only minimize the square error between the estimated in-cylinder pressure and the measured in-cylinder pressure (teacher data) during learning, so they do not monitor whether the "engine sound from which noise has been removed" is one from which only noise has been removed satisfactorily. For example, the conventional technique described in Patent Document 3 applies a noise removal mask α that does not take into account phase components related to the engine vicinity sound (input physical quantity) to the engine vicinity sound to extract the "engine sound from which noise has been removed," i.e., the knocking sound (corresponding to the "extracted signal" in this embodiment). At that time, the conventional technique described in Patent Document 3 does not use a function for measuring the degree of separation of sounds, so there is a possibility that the knocking sound (target sound) that should not be removed may be removed from the engine vicinity sound together with the noise. Therefore, the conventional technique described in Patent Document 3 may degrade the performance of evaluating the presence or absence of knocking.

[0099] In contrast, the signal processing device 10 according to the present embodiment is configured to evaluate whether or not a knocking sound (target sound) is mixed in the noise removed from the sound near the engine (input physical quantity) during learning. To achieve this, the signal processing device 10 according to the present embodiment is configured to learn so that the correlation between the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced. Specifically, as shown in FIG. 3A, learning is performed so that the coherence between the first signal Sfi acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced. In addition, the signal processing device 10 according to the present embodiment is configured to learn so that the correlation between the extracted knocking sound 91a (extracted signal) and the teacher signal (observed knocking in-cylinder pressure 93) is increased. 3A, learning is performed so that the coherence between the second signal Sse acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted knocking sound 91a (extracted signal) and the teacher signal (observed knocking in-cylinder pressure 93) becomes large. The signal processing device 10 according to this embodiment can improve the evaluation performance of the presence or absence of knocking compared to the conventional techniques described in Patent Documents 2 and 3.

[0100] Fig. 8A shows a spectrogram F11 of the noise 91b (noise component), a signal waveform F12 of the first signal Sfi (Fig. 3A) obtained by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram F11, and a signal waveform F93 of the observed knocking in-cylinder pressure 93 (teacher signal). Fig. 8A also shows coherence F13 that is minimized in the process of learning the signal waveform F12 of the first signal Sfi (Fig. 3A) and the signal waveform F93 of the observed knocking in-cylinder pressure 93 (teacher signal).

[0101] Fig. 8B shows a spectrogram F21 of the extracted knocking sound 91a (extracted signal), a signal waveform F22 of the second signal Sse (Fig. 3A) obtained by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram F21, and a signal waveform F93 of the observed knocking in-cylinder pressure 93 (teacher signal). Fig. 8B also shows coherence F23 that is maximized in the learning process between the signal waveform F22 of the second signal Sse (Fig. 3A) and the signal waveform F93 of the observed knocking in-cylinder pressure 93 (teacher signal).

[0102] Coherence is defined by the following equation (1): Coherence function γ 2 indicates the degree of correlation between the input and output of the system. The coherence function γ 2 is calculated by dividing the square of the absolute value of the cross spectrum by the power spectrum of the measurement input and the system output, as shown in the following equation (1).

number

[0103] Here, Wxy is the average value of the cross spectrum, Wxx is the average value of the power spectrum of x, and Wyy is the average value of the power spectrum of y. Coherence function γ 2 takes values ​​between 0 and 1. γ 2 When (f) is 1, it indicates that the system output at that frequency f is entirely due to the measured input. Also, γ 2 If (f) is 0, it means that the output of the system is completely independent of the measured input at that frequency f. Also, 0<γ 2 If (f)<1, it is likely that there is a signal unrelated to the measurement, noise occurring within the system, nonlinearity of the system, etc.

[0104] Alternatively, the coherent output power may be used as a method for evaluating whether or not a knocking sound (target sound) is mixed into the noise removed from the sound near the engine (input physical quantity). The learning unit 21 learns the weight W and transfer function H of the neural network that generates the noise removal mask α so that the coherence or coherent output power between the signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced, and the coherence or coherent output power between the signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on the extracted signal (extracted knocking sound 91a) and the teacher signal (observed knocking in-cylinder pressure 93) is increased. This allows the signal processing device 10 to further improve the evaluation performance for the presence or absence of knocking.

[0105] In addition, for coherence and coherent output power, when the evaluation objects are the following two sets of "I" and "II" and the evaluation methods are the following two sets of "A" and "B", the signal processing device 10 enables evaluation of the following four evaluation patterns. (Subject to evaluation) I: Noise components and teacher signals II: Extracted signals and teacher signals (Evaluation method) A: Coherence B: Coherent output power (Evaluation pattern) 1: Evaluate I with an A, and II with an A. 2: Evaluate I with B, Evaluate II with B. 3: Evaluate I with A and II with B. 4: Evaluate I with a B and II with an A.

[0106] The coherent output power is given by the coherence function γ 2 and the average value Wyy of the power spectrum of y. The coherent output power is the power that is contained in the output of the system due to the input.

[0107] 3A, the learning unit 21 learns the weights W and transfer function H of the neural network that generates the noise removal mask α, using the neural network 94, so that the error between a spectrogram obtained by performing a short-time Fourier transform (STFT) on the observed knocking in-cylinder pressure 93 (teacher signal) and the estimated knocking in-cylinder pressure 92 (estimated signal) is minimized. This enables the signal processing device 10 to improve the learning accuracy of the weights W and transfer function H of the neural network that generates the noise removal mask α.

[0108] Furthermore, the learning unit 21 may perform an inverse short-time Fourier transform (ISTFT) and a fast Fourier transform (FFT) on the observed knocking in-cylinder pressure 93 (teacher signal) and the extracted signal (extracted knocking sound 91a), multiply the result by a transfer function H, and use the neural network 94 to learn the weight W of the neural network that generates the noise removal mask α and the transfer function H so that an error between the signal waveform of the estimated knocking in-cylinder pressure 92 (estimated signal) obtained by performing an inverse fast Fourier transform (IFFT) is minimized. This enables the signal processing device 10 to further improve the learning accuracy of the weight W and the transfer function H of the neural network that generates the noise removal mask α.

[0109] FIG. 8C shows an example of converting the extracted knocking sound 91a (extracted signal) into an estimated knocking in-cylinder pressure 92 (estimated signal). FIG. 8C shows a spectrogram F31 of the extracted knocking sound 91a (extracted signal), a signal waveform F32 obtained by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram F31, and a spectrum F33 obtained by further performing a fast Fourier transform (FFT). FIG. 8C also shows a spectrum F35 of the estimated knocking in-cylinder pressure obtained by multiplying the signal of the spectrum F33 by a transfer function H. Note that FIG. 8C shows a frequency response characteristic F34 indicating the correspondence between the frequency and the coefficient as an example of the transfer function H. FIG. 8C also shows a signal waveform F36 obtained by performing an inverse fast Fourier transform (IFFT) on the spectrum F35, and a spectrogram F37 of the estimated knocking in-cylinder pressure 92 (estimated signal) obtained by further performing a short-time Fourier transform (STFT).

[0110] In this embodiment, the learning unit 21 learns weights W of a neural network that generates a noise removal mask α using a convolutional neural network (CNN). Furthermore, the learning unit 21 learns weights to be multiplied by the spectrum F33 of the extracted signal as a transfer function H. Note that the learning unit 21 is not limited to a CNN, and may have other configurations such as a recurrent neural network (RNN) or a Transformer, or may use other machine learning methods.

[0111] As shown in FIG. 9, U-Net95 is an example of a neural network 94 that learns the weights W of the neural network that generates the noise removal mask α. This U-Net95 is a type of Encoder-Decoder model, and is a deep learning method used for image recognition and sound separation. In sound separation, U-Net95 performs convolution (stride is 2 or more) in the downward pass 96 (Encoder), and extracts sound features as the layer 97 becomes deeper. Meanwhile, in the upward pass 98 (Decoder), the noise removal mask α is generated by performing deconvolution and UP sampling (dilation) from the extracted sound features. Up to this point, the configuration of a general Encoder-Decoder model. Furthermore, in U-Net95, the output 99 from the convolution layer of each Encoder is merged into the convolution layer of the Decoder. As a result, U-Net95 can generate a noise removal mask α with higher accuracy than a general Encoder-Decoder model. The signal processing device 10 can obtain an extracted signal (extracted knocking sound 91a) by generating a noise removal mask α and multiplying it by the near-engine sound 90. The noise removal mask α is a spatial mask represented by frequency and time.

[0112] In this way, the learning unit 21 learns the weights W of the neural network that generates the noise removal mask α using U-Net 95, so that the trained neural network generates an appropriate noise removal mask α according to the input sound in the vicinity of the engine 1. Therefore, the signal processing device 10 can accurately estimate the knocking in-cylinder pressure of the engine 1 and satisfactorily extract the knocking sound of the engine 1.

[0113] <Influence of phase components> The influence of phase components will be described with reference to Figures 10A and 10B. Figure 10A is an explanatory diagram of a calculation example in the case where phase components of pressure, vibration, sound, etc. are not taken into consideration. Figure 10B is an explanatory diagram of a calculation example in the case where phase components of pressure, vibration, sound, etc. are taken into consideration. In Figures 10A and 10B, the arrows in the circles represent phase components contained in pressure, vibration, sound, etc.

[0114] In both the examples shown in FIGS. 10A and 10B, the following conditions are assumed: The system is a linear time-invariant system (a system in which the transfer characteristics between the input and output are linear and do not change with time).

[0115] 10A, when the engine 1 is driven, an observed knocking in-cylinder pressure 93 is observed by the in-cylinder pressure sensor 5 (FIG. 1). The observed knocking in-cylinder pressure 93 propagates through the engine 1, causing a knocking sound 82 to be emitted. Furthermore, the knocking sound 82 is combined with mechanical noise 83 (noise component), which is a background sound, and a near-engine sound 90 (input physical quantity) is observed by the sound pressure sensor 4.

[0116] FIG. 10A shows an example of rough calculation of the observed knocking in-cylinder pressure 93, knocking vibration 81, knocking sound 82, mechanical noise 83, and near-engine sound 90 as follows, assuming that the phase φ does not change. Observed knocking cylinder pressure 93 = ASin(Ωt+φ) Knocking vibration 81 = ABSin(Ωt+φ) Knocking sound 82 = ABCSin(Ωt+φ) Mechanical noise 83 = NSin(Ωt+φN) Near-engine noise 90 = (ABC+N)Sin(Ωt+φ)

[0117] 10B is an example in which the phase is taken into consideration. As shown below, an example of a rough calculation of the observed knocking in-cylinder pressure 93, the knocking vibration 81, the knocking sound 82, the mechanical noise 83, and the sound near the engine 90 is shown, and it is taken into consideration that the phase φ changes every time the dimension (unit) changes. Observed knocking cylinder pressure 93 = ASin(Ωt+φA) Knocking vibration 81 = ABSin(Ωt+φA+φB) Knocking sound 82 = ABCSin(Ωt+φA+φB+φC) Mechanical noise 83 = NSin(Ωt+φN) Near-engine noise 90 = ABCSin(Ωt+φA+φB+φC)+NSin(Ωt+φN)

[0118] In the example shown in FIG. 10B, the change in the phase component when the pressure becomes vibration and is emitted as sound is taken into consideration, so that the signal can be analyzed better than the example shown in FIG. 10A. Therefore, as shown in FIG. 3A, the signal processing device 10 is configured to learn the weight W and the transfer function H of the neural network that generates the noise removal mask α while taking the phase into consideration. Such a signal processing device 10 can effectively separate the engine vicinity sound 90 (input physical quantity) into the extracted knocking sound 91a (extracted signal) and the noise 91b (noise component). In other words, the signal processing device 10 can effectively separate the engine vicinity sound 90 into the extracted knocking sound 91a and the noise 91b so that the knocking sound (target sound) is not mixed into the noise. Such a signal processing device 10 can improve the evaluation performance of the presence or absence of knocking compared to the conventional techniques described in Patent Documents 2 and 3. In addition, such a signal processing device 10 can perform a good sensory test.

[0119] (Overview of sensory testing) An overview of the sensory test will be described with reference to Figures 11A to 12C. Figures 11A to 12C are explanatory diagrams of signal processing in the sensory test.

[0120] 2, pressure and sound signals are collected by a data collecting device 6 and output to a signal processing device 10. In the signal processing device 10, a signal extracting unit 11 extracts each signal, a spectrogram calculating unit 12 calculates a spectrogram, and a signal storage unit 13 stores the spectrogram.

[0121] The signal processing device 10 according to this embodiment can execute the separation mode and the sensory test mode. In the separation mode, the signal processing device 10 separates the engine vicinity sound 90 (input physical quantity) into an extracted knocking sound 91a (extracted signal) and noise 91b (noise component). In the sensory test mode, the signal processing device 10 changes the level of the extracted knocking sound 91a to generate a level-changed extracted signal (level-changed knocking sound 91aa (FIG. 12A)), and combines it with the noise 91b to generate a processed sound 91c (FIG. 12C).

[0122] In the sensory test mode, the examiner wears headphones 8 (FIG. 1) on his / her head. Then, the examiner or an operator who separately specifies the level announces an acceptable level to those around him / her and operates a level specification unit 9 to input level specification information to the signal processing device 10. The signal processing device 10 changes the level of the extracted knocking sound 91a based on the input level specification information.

[0123] FIG. 11A shows the engine vicinity sound 90 (input physical quantity), and FIG. 11B and FIG. 11C show the extracted knocking sound 91a (extracted signal) and noise 91b (noise component) separated from the engine vicinity sound 90 in the separation mode. FIG. 12A shows the level-changed knocking sound 91aa (level-changed extracted signal) generated by changing the level of the extracted knocking sound 91a (extracted signal) to be increased by 3 dB in the sensory test mode, FIG. 12B shows the noise 91b to be synthesized with the level-changed knocking sound 91aa, and FIG. 12C shows the processed sound 91c obtained by synthesizing the level-changed knocking sound 91aa and the noise 91b. The signal processing device 10 can create a state in which the target sound (knocking sound) to be inspected is easily distinguished by generating the processed sound 91c. Such a signal processing device 10 can allow the inspector to grasp the presence or absence of the target sound (knocking sound) with high accuracy, thereby improving the inspection performance. Furthermore, the signal processing device 10 calculates the sum of the absolute values ​​of the spectrograms of the processed sounds 91c that are determined to be unacceptable by the inspector, and writes an arbitrary value equal to or smaller than the sum in the threshold storage unit 25. This makes the threshold value T a value that matches the senses of the inspector.

[0124] Depending on the operation, the signal processing device 10 may multiply the level-changed extraction signal (level-changed knocking sound 91aa) by a transfer function He (not shown) from the measurement object to the position of the listener's ears during the sensory test mode, and then synthesize a noise component at the position of the listener's ears to generate a processed sound.

[0125] <Operation of the signal processing device (estimation device)> The operation of the signal processing device 10 will be described with reference to Figs. 13 to 18. Fig. 13 is a flowchart showing the data collection process of the signal processing device 10. Fig. 14A is a flowchart showing the learning process of the signal processing device 10. Fig. 14B is a flowchart showing a subroutine of the learning process. In the learning mode, the signal processing device 10 performs the data collection process of Fig. 13, and then performs the learning process of Figs. 14A and 14B. Fig. 14C is a flowchart showing a modified example of the subroutine of the learning process. Fig. 15A is a flowchart showing the threshold calculation process of the signal processing device 10. In the threshold calculation mode, the signal processing device 10 performs the data collection process of Fig. 13, and then performs the threshold calculation process of Fig. 15A. Fig. 15B is a flowchart showing the threshold calculation process. In the case where the separation process is performed in the separation mode and the sensory test process is performed in the sensory test mode, the signal processing device 10 performs the threshold calculation process of Fig. 15B. Fig. 16 is a flowchart showing the judgment process of the signal processing device 10. In the judgment mode, the signal processing device 10 performs the data collection process of Fig. 13, and then performs the judgment process of Fig. 16. However, the signal processing device 10 may perform the judgment process of Fig. 16 in real time while performing the data collection process of Fig. 13. Fig. 17 is a flowchart showing the separation process of the signal processing device 10. In the separation mode, the signal processing device 10 performs the data collection process of Fig. 13, and then performs the separation process of Fig. 17. Fig. 18 is a flowchart showing the sensory test process of the signal processing device 10. In the sensory test mode, the signal processing device 10 performs the sensory test process of Fig. 18 before performing the threshold calculation process of Fig. 15B.

[0126] (Data collection and processing) The data collection process will now be described with reference to FIG. 13, in step S20, the data collecting device 6 inputs the sound pressure signal, the in-cylinder pressure signal (teacher signal), and the angle information to the signal extracting unit 11. In addition, in the case of the separation mode, the threshold calculation mode, or the determination mode, it is not necessary to input the in-cylinder pressure signal in step S20. In step S21, the signal extraction unit 11 calculates the combustion stroke timing of each cylinder from the angle information.

[0127] In step S22, the signal extracting unit 11 extracts the sound pressure signal and the in-cylinder pressure signal near the TDC in accordance with the combustion stroke timing calculated in step S21. In step S23, the spectrogram calculation unit 12 performs a short-time Fourier transform (STFT) on the sound pressure signal extracted in step S22 to calculate a spectrogram of the sound pressure signal. In step S24, the spectrogram calculation unit 12 writes the spectrogram of the sound pressure signal in the signal storage unit 13. Furthermore, the signal extraction unit 11 writes the in-cylinder pressure signal (teacher signal) in the signal storage unit 13 if an in-cylinder pressure signal is present.

[0128] (Learning process) The learning process executed in the learning mode will be described with reference to FIGS. 14A to 14C. As shown in FIG. 14A, in step S30, the signal processing device 10 reads out the spectrogram of the sound pressure signal and the in-cylinder pressure signal from the signal storage unit 13, and inputs them to the learning unit 21. In step S31, the learning unit 21 uses the spectrogram of the sound pressure signal and the in-cylinder pressure signal input in step S30 to learn weights W and transfer function H of a neural network that generates a noise removal mask α by taking into account amplitude and phase components related to the input physical quantity in the neural network 94. The learning unit 21 learns weights W of a neural network that generates a noise removal mask α using a convolutional neural network (e.g., U-Net), and learns a weight to be multiplied by the spectrum of the extracted signal as transfer function H.

[0129] In step S32, the learning unit 21 writes, into the learned parameter storage unit 22, the weights W and the transfer function H of the neural network that generates the learned noise removal mask α.

[0130] The process of step S31 is performed by, for example, a series of processes shown in FIG. 14B. As shown in FIG. 14B, in step S31a, the learning processing unit 20 generates a noise removal mask α by the neural network 94. In step S101, the learning unit 21 separates the input physical quantity (the engine vicinity sound 90) into a noise component (noise 91b) and an extracted signal (extracted knocking sound 91a) using the noise removal mask α generated in step S31a.

[0131] In step S102, the learning unit 21 multiplies the transfer function H to convert the extracted signal (extracted knocking sound 91a) into an estimated signal (estimated knocking in-cylinder pressure 92). In step S103, the learning unit 21 updates the weight W and the transfer function H of the neural network that generates the noise removal mask α. In step S103, the learning unit 21 updates the weight W and the transfer function H of the neural network that generates the noise removal mask α so that the coherence between the first signal Sfi (FIG. 3A) acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced, and the coherence between the second signal Sse (FIG. 3A) acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted signal (extracted knocking sound 91a) and the teacher signal (observed knocking in-cylinder pressure 93) is increased, and the error between the estimated signal (estimated knocking in-cylinder pressure 92) and the spectrogram obtained by performing a short-time Fourier transform (STFT) on the teacher signal (observed knocking in-cylinder pressure 93) is reduced.

[0132] In step S104, the learning unit 21 determines whether the coherence and error have converged, and if it is determined that they have converged (if "Yes"), it terminates the processing of step S31, whereas if it is determined that they have not converged (if "No"), it performs the processing of step S105.

[0133] In step S105, the learning unit 21 generates the noise reduction input physical quantity 90A, and shuffles the near-engine sound 90 and the noise reduction input physical quantity 90A. After step S105, the learning unit 21 repeats the processes in and after step S31a. In step S104, the state in which the coherence and error have converged is a state in which the coherence between the teacher signal (observed knocking in-cylinder pressure 93) and the signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on the noise component (noise 91b) is a value close to "0", and the coherence between the teacher signal (observed knocking in-cylinder pressure 93) and the signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on the extracted signal (extracted knocking sound 91a) is a value close to "1", and the error of the estimated signal (estimated knocking in-cylinder pressure 92) with respect to the spectrogram obtained by performing a short-time Fourier transform (STFT) on the teacher signal (observed knocking in-cylinder pressure 93) has converged to a value close to "0".

[0134] Note that the process of step S31 may be, for example, the process shown in FIG. 14C, instead of the process shown in FIG. 14B. The process shown in FIG. 14C differs from the process shown in FIG. 14B in that the process in step S103a is performed instead of the process in step S103. In step S103a, in addition to the conditions in step S103, the learning unit 21 calculates the maximum value and the difference between the maximum value and the minimum value (knocking intensity) of the signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on the estimated signal (estimated knocking in-cylinder pressure 92), and the maximum value and the difference between the maximum value and the minimum value (knocking intensity) of the teacher signal (observed knocking in-cylinder pressure 93), and updates the weight W and transfer function H of the neural network that generates the noise removal mask α so as to satisfy the condition that the error between the former maximum value and the latter maximum value, and the error between the former maximum value and the minimum value (knocking intensity) and the difference between the latter maximum value and the minimum value (knocking intensity) are each small. That is, in step S103a, the learning unit 21 performs an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise component (noise 91b) to reduce coherence between the first signal Sfi (FIG. 3A) and the teacher signal (observed knocking in-cylinder pressure 93), and performs an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted signal (extracted knocking sound 91a) to increase coherence between the second signal Sse (FIG. 3A) and the teacher signal (observed knocking in-cylinder pressure 93), and performs a short-time Fourier transform (STFT) on the estimated signal (estimated knocking in-cylinder pressure 92) and the teacher signal (observed knocking in-cylinder pressure 93). Further, an inverse short-time Fourier transform (ISTFT) is performed on the estimated signal (estimated knocking in-cylinder pressure 92) to calculate the maximum value of the signal waveform, the difference between the maximum value and the minimum value (knocking intensity), and the maximum value and the difference between the maximum value and the minimum value (knocking intensity) of the teacher signal (observed knocking in-cylinder pressure 93). The weight W and transfer function H of the neural network that generates the noise removal mask α are updated so that the error between the former maximum value and the latter maximum value, and the error between the former maximum value and the minimum value (knocking intensity) and the difference between the latter maximum value and the minimum value (knocking intensity) are each reduced.

[0135] 14B and 14C, step S105 may be performed the first time (after step S31). Also, the flow shown in Fig. 14B and 14C may be configured to perform the first process without step S105, and then perform the second process including step S105 after learning is completed.

[0136] (Threshold calculation process) The threshold calculation process executed in the threshold calculation mode will be described with reference to FIG. 15A. As shown in FIG. 15A, in step S40, the learning processing unit 20 generates a noise removal mask α by the neural network 94. In step S41, the first estimation unit 23 uses the noise removal mask α generated by the neural network 94 to extract the knocking sound of the engine 1 from the spectrogram of the sound pressure signal. In step S42, the threshold calculation unit 24 sums up the absolute values ​​of the spectrograms of the knocking sound in each cycle of the engine 1.

[0137] In step S43, the threshold calculation unit 24 calculates the median value of the spectrogram of the knocking sound in all cycles of the engine 1. In step S44, the threshold calculation unit 24 adds an arbitrary margin to the median value to obtain a threshold value. In step S45, the threshold calculation unit 24 writes the calculated threshold into the threshold storage unit 25.

[0138] The signal processing device 10 has a function of executing the threshold calculation process of Fig. 15B after the separation process of Fig. 17 and the sensory test process of Fig. 18. Hereinafter, the threshold calculation process performed after the separation process of Fig. 17 and the sensory test process of Fig. 18 will be described with reference to Fig. 15B. As shown in FIG. 15B, in step S140, the signal adjustment unit 51 acquires signals that are unacceptable in the sensory test from the noise component storage unit 26c and the extracted signal storage unit 26d, and supplies them to the first estimating unit 23. In step S141, the learning processing unit 20 generates a noise removal mask α by the neural network 94. In step S142, the first estimation unit 23 uses the noise removal mask α generated by the neural network 94 to obtain a spectrogram of the knocking sound from the spectrogram of the signal that is unacceptable in the sensory test, and supplies the spectrogram to the threshold calculation unit 24. In step S143, the threshold calculation unit 24 sums up the absolute values ​​of the spectrogram of the knocking sound supplied from the first estimating unit 23.

[0139] In step S144, the threshold calculation unit 24 sets an arbitrary value equal to or smaller than the total sum value as the threshold. In step S145, the threshold calculation unit 24 writes the calculated threshold into the threshold storage unit 25.

[0140] (Decision making process) The determination process executed in the determination mode will be described with reference to FIG. As shown in FIG. 16, in step S50, the learning processing unit 20 generates a noise removal mask α by the neural network 94. In step S51, the second estimation unit 31 extracts the knocking sound of the engine 1 from the spectrogram of the sound pressure signal input from the signal storage unit 13, using the noise removal mask α generated by the neural network 94.

[0141] In step S52, the threshold determination unit 32 sums up the absolute values ​​of the spectrograms of the knocking sound in each cycle of the engine 1. In step S53, the threshold determination unit 32 compares the sum of the knocking noise with the threshold value already stored in the threshold value storage unit 25, and determines whether the knocking noise exceeds the threshold value. If the sum of the knocking noises exceeds the threshold value (Yes in step S53), the threshold value determination unit 32 determines that knocking has occurred (step S54).

[0142] If the sum of the knocking noises is equal to or smaller than the threshold value (No in step S53), the threshold value determination unit 32 determines that there is no knocking (step S55). In step S56, the threshold determination unit 32 outputs the result of the threshold determination and the knocking sound to the monitor 7.

[0143] (Separation processing) The separation process performed in the separation mode will now be described with reference to FIG. As shown in FIG. 17, in step S60, the learning processing unit 20 generates a noise removal mask α by the neural network 94. In step S61, the separator 40 uses the noise removal mask α generated by the neural network 94 to separate the input physical quantity into a noise component and an extracted signal. In step S62, the learning processing unit 20 stores the separated noise components and extracted signals in the corresponding noise component storage unit 26c and extracted signal storage unit 26d, respectively.

[0144] (Sensory test treatment) The sensory test process executed in the sensory test mode will be described with reference to FIG. As shown in FIG. 18, the signal adjustment unit 51 of the signal synthesis unit 50 receives the input of level designation information from the level designation unit 9 in step S70. In step S71, the signal adjustment unit 51 acquires the extraction signal from the extraction signal storage unit 26d and the noise component from the noise component storage unit 26c, and changes the level of the extraction signal in accordance with the level designation information to generate a level-changed extraction signal.

[0145] In step S72, the signal adjustment unit 51 synthesizes the level-changed extraction signal and the noise component to generate a processed sound. In step S73, the signal output unit 52 of the signal synthesis unit 50 outputs the processed sound to the sound emitting unit (headphones 8) to cause the sound emitting unit to emit the processed sound. In step S74, the signal synthesis unit 50 determines whether or not an instruction to end has been given from the level designation unit 9, and if it is determined that an instruction to end has been given (if "Yes"), it outputs the processed sound to the first estimation unit 23, calculates a threshold value by the threshold calculation unit 24, writes it into the threshold storage unit 25, and ends the processing of Fig. 18. On the other hand, if it is determined that an instruction to end has not been given (if "No"), in step S75, the signal adjustment unit 51 of the signal synthesis unit 50 accepts a change in the level designation information from the level designation unit 9, and repeats the processing from step S71 onwards.

[0146] After the separation process in FIG. 17 and the sensory test process in FIG. 18, the signal processing device 10 performs a threshold calculation process shown in FIG. 15B on signals that are unacceptable in the sensory test (FIG. 18).

[0147] In this embodiment, the case where coherence is used as an element representing the relevance has been described, but coherent output power may be used as an element representing the relevance.

[0148] <Noise reduction input physical quantity generation function and shuffle function> The signal processing device 10 according to the first embodiment has a noise-reduced input physical quantity generation function and a shuffle function in the learning mode. The noise-reduced input physical quantity generation function is a function for generating a noise-reduced input physical quantity 90A having less noise than the input physical quantity (the engine nearby sound 90 (FIG. 3A)). The shuffle function is a function for shuffling the input physical quantity (the engine nearby sound 90 (FIG. 3A)) and the noise-reduced input physical quantity 90A.

[0149] The noise reduction input physical quantity generation function and the shuffle function will be described below. In order to easily explain the noise reduction input physical quantity generation function and the shuffle function, the configuration and operation of a signal processing device 10Z according to a comparative example will be described, and then the operation of the signal processing device 10 according to the first embodiment will be described. The signal processing device 10Z according to the comparative example is a device that does not have the noise reduction input physical quantity generation function and the shuffle function.

[0150] First, the configuration and operation of a signal processing device 10Z according to a comparative example will be described with reference to Fig. 19A to Fig. 19C. Fig. 19A is a block diagram showing the configuration of the signal processing device 10Z according to the comparative example. Fig. 19B is an explanatory diagram of the learning mode of the signal processing device 10Z according to the comparative example. Fig. 19C is an explanatory diagram of the operation of the signal processing device 10Z according to the comparative example in the learning mode.

[0151] The signal processing device 10Z in FIG. 19A is an embodiment of a signal processing system 100Z of a comparative example that is not equipped with a noise reduction input physical quantity generating function and a shuffle function, unlike the signal processing system 100 shown in FIG. 1. As shown in FIG. 19A, the signal processing device 10Z of the comparative example is different from the signal processing device 10 (see FIG. 2) according to the first embodiment in that it has a learning processing unit 20Z instead of the learning processing unit 20 (see FIG. 2). The learning processing unit 20Z is different from the learning processing unit 20 (see FIG. 2) in that it does not have the noise reduction input physical quantity generating unit 15 and the selecting unit 16, and has a learning unit 21Z instead of the learning unit 21 (see FIG. 2). The learning unit 21Z is different from the learning unit 21 (see FIG. 2) in that it learns weights of a neural network 94 that generates a noise removal mask α for removing a noise component (noise 91b (FIG. 19B)) from an input physical quantity (near-engine sound 90 (FIG. 19B)) in a learning mode.

[0152] As shown in Fig. 19B, in the learning mode, in the signal processing device 10Z according to the comparative example, the near-engine sound 90 is supplied to the learning unit 21Z. In response to this, the learning unit 21Z of the signal processing device 10Z separates the near-engine sound 90 into noise 91b (noise component) and an extracted knocking sound 91a (extracted signal). At that time, the learning unit 21Z of the signal processing device 10Z multiplies the near-engine sound 90 by a noise removal mask α to obtain the extracted knocking sound 91a (extracted signal). In addition, the learning unit 21Z of the signal processing device 10Z obtains the noise 91b (noise component) by subtracting the extracted knocking sound 91a (extracted signal) from the near-engine sound 90.

[0153] The learning unit 21Z of the signal processing device 10Z learns the weight W of the neural network that generates the noise removal mask α, and the transfer function H for converting the extracted knocking sound 91a (extracted signal) into an estimated knocking in-cylinder pressure 92 (estimated signal) of the same dimension (unit) as the observed knocking in-cylinder pressure 93 (teacher signal) while taking into account the phase. At this time, in the comparative example, the learning unit 21Z of the signal processing device 10Z performs an inverse short-time Fourier transform (ISTFT) and a fast Fourier transform (FFT) on the extracted knocking sound 91a (extracted signal), multiplies it by the transfer function H, and performs an inverse fast Fourier transform (IFFT) and a short-time Fourier transform (STFT) to convert the extracted knocking sound 91a (extracted signal) into the estimated knocking in-cylinder pressure 92 (estimated signal). Note that the transfer function H is an amplitude (gain) and a phase component for converting the extracted knocking sound 91a (extracted signal) into the estimated knocking in-cylinder pressure 92 (estimated signal).

[0154] In addition, the learning unit 21Z of the signal processing device 10Z learns the weights of the neural network so that the correlation between the first signal Sfi obtained by performing arbitrary processing on the noise 91b (noise component) and the teacher signal (observed knocking in-cylinder pressure 93) becomes smaller, and the correlation between the second signal Sse obtained by performing arbitrary processing on the extracted knocking sound 91a (extracted signal) and the teacher signal (observed knocking in-cylinder pressure 93) becomes larger. Specifically, the learning unit 21Z learns the weight W and transfer function H of the neural network that generates the noise removal mask α so that the coherence between the first signal Sfi acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise 91b (noise component) and the observed knocking in-cylinder pressure 93 (teacher signal) is reduced, and the coherence between the second signal Sse acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted knocking sound 91a (extraction signal) and the observed knocking in-cylinder pressure 93 (teacher signal) is increased. This enables the signal processing device 10Z to remove the sound caused by the in-cylinder pressure from the noise 91b.

[0155] Fig. 19C shows the operation of the signal processing device 10Z according to the comparative example in the learning mode. The bold frame in Fig. 19C shows the components that operate in the learning mode. The bold arrow in Fig. 19C shows the signals that are output in the learning mode.

[0156] As shown in FIG. 19C, during the learning mode, the signal processing device 10Z of the comparative example connects the switch 14 to the learning mode connection unit M1 and outputs the spectrogram of the near-engine sound 90 and the observed knocking in-cylinder pressure 93 (teacher signal) stored in the signal memory unit 13 to the learning unit 21Z.

[0157] In response to this, the learning unit 21Z of the signal processing device 10Z learns the weighting W of a neural network 94 (FIG. 19B) that generates a noise removal mask α that removes noise 91b (noise component) from the near-engine sound 90, and a transfer function H that converts the extracted knocking sound 91a (extracted signal) extracted by the generated noise removal mask α into an estimated knocking in-cylinder pressure 92 (estimated signal) of the engine 1 at the time when knocking occurs. At this time, the learning unit 21Z of the signal processing device 10Z generates the noise removal mask α corresponding to the near-engine sound 90 every time a new near-engine sound 90 is input.

[0158] Then, the learning unit 21Z of the signal processing device 10Z stores the learned parameters (weight W and transfer function H of the neural network that generates the noise removal mask α) in the learned parameter storage unit 22. In addition, the learning unit 21Z of the signal processing device 10Z stores the observed knocking in-cylinder pressure 93 (teacher signal) in the teacher signal storage unit 26a, and stores the estimated knocking in-cylinder pressure 92 (estimated signal) in the estimated signal storage unit 26b.

[0159] <Example of change of use> Incidentally, the signal processing device 10 according to the present embodiment can be used for various applications, not limited to the above-mentioned applications. For example, the signal processing device 10 according to the present embodiment can be used for applications in which the verification target is one of a plurality of electronic components mounted on an electronic device, and the type of operation sound generated by the electronic component to be verified is verified. In this case, the input physical quantity is the sound near the electronic device on which the electronic component is mounted. The sound near the electronic device includes the transient operation sounds of all electronic components mounted on the electronic device, and sounds on which other noises are superimposed, and the operation sound of the electronic component to be verified is a part of the electronic components. The noise components include the operation sounds of the other electronic components other than the electronic component to be verified, among the electronic components mounted on the electronic device, and other noises. The sound near the electronic device is a signal in which the transient noise components are large, and the target sound is more likely to be missed than the sound near the engine 90. In addition, the sound near the electronic device is a signal in which the missed sound is more noticeable because the stationary noise components are small.

[0160] Hereinafter, the overall configuration of the signal processing system 100 suitable for such an example of change of use will be described with reference to Fig. 20, and further, the processing in the learning mode of the signal processing system 100 suitable for the signal processing device 10 in such an example of change of use will be described with reference to Fig. 21. Fig. 20 is a block diagram showing the overall configuration of the signal processing system 100 for realizing the example of change of use of the signal processing device 10. Fig. 21 is an explanatory diagram of the learning mode in the signal processing device 10 whose use has been changed.

[0161] As shown in Fig. 20, the signal processing device 10 according to this embodiment can be used to verify what kind of operation sound is generated by an electronic component 101 that is one of a plurality of electronic components 101, 102 mounted on an electronic device 103 as a verification target. In this case, the input physical quantity is the sound near the electronic device. The sound near the electronic device includes transient operation sounds of the electronic components 101, 102 mounted on the electronic device 103 and sounds on which other noises are superimposed, and the operation sound of the electronic component 101 to be verified is a part of the sound. The noise components include operation sounds of the other electronic components 102 that are not the verification target and other noises.

[0162] 20, the signal processing system 100 includes a sound pressure sensor 4, a data collecting device 6, a monitor 7, headphones 8, a level designation unit 9, and a signal processing device 10, similar to those in Fig. 1, and includes an acceleration sensor 105 instead of the in-cylinder pressure sensor 5 in Fig. 1. The sound pressure sensor 4 acquires the operating sounds of the electronic device 103, including the operating sounds of all electronic components 101, 102 mounted on the electronic device 103 and other noises, converts them into sound pressure signals, and outputs them to the data collecting device 6.

[0163] Acceleration sensor 105 is attached so as to be in contact with electronic component 101 to be verified, and acquires the vibration acceleration of electronic component 101 to be verified, converts it into a vibration acceleration signal, and outputs it to data collection device 6. Here, when electronic component 101 is not generating an operating sound, the vibration acceleration signal output from acceleration sensor 105 does not include a component correlated with the operating sound. On the other hand, when electronic component 101 is generating an operating sound, the vibration acceleration signal output from acceleration sensor 105 includes a component correlated with the operating sound.

[0164] The data collection device 6 A / D converts the sound pressure signal of the electronic device 103 and the vibration acceleration signal of the electronic component 101 to be verified, and outputs them to the signal processing device 10. The signal processing device 10 learns the weight W (FIG. 21) and the transfer function H (FIG. 21) of the neural network 94 (FIG. 21) that generates the noise removal mask α. The monitor 7, the headphones 8, the level designation unit 9, and the signal processing device 10 have been described above with reference to FIG. 1, so their explanation will be omitted here.

[0165] In other words, the sound near the engine in the processing of the engine knocking sound corresponds to the sound at the position of the listener's ear in the processing of the operation sound of the electronic device 103, and therefore corresponds to the sound near the electronic device A90. The extracted knocking sound in the processing of the engine knocking sound corresponds to the operation sound of the electronic component 101 to be verified. The estimated knocking in-cylinder pressure in the processing of the engine knocking sound corresponds to an estimated value of the vibration acceleration of the electronic component 101 to be verified. The observed knocking in-cylinder pressure in the processing of the engine knocking sound corresponds to an observed value of the vibration acceleration of the electronic component 101 to be verified. In other words, the teacher signal in the processing of the operation sound of the electronic component 101 is the observed value of the vibration acceleration of the electronic component 101 to be verified.

[0166] When the signal processing device 10 of Figure 1 is used in this changed use example, the operation of the signal processing device 10 in the learning mode of the changed use example shown in Figure 21 differs from the operation of the signal processing device 10 in the learning mode of the changed use example shown in Figure 3A in the following ways. (1) In place of the engine nearby sound 90 (FIG. 3A), an electronic device nearby sound A90 is input to the noise reduction input physical quantity generating unit 15 and the selecting unit 16 as an input physical quantity. (2) The noise-reduced input physical quantity generation unit 15 generates a noise-reduced input physical quantity A90A in which the noise components (in the example of the changed use, noise A91b) contained in the input physical quantity (in the example of the changed use, sound A90 near the electronic device) are reduced, instead of the noise-reduced input physical quantity 90A (FIG. 3A). (3) The selection unit 16 selects either or both of the electronic device vicinity sound A90 and the noise reduction input physical quantity A90A, which are input physical quantities, and supplies them to the learning unit 21. (4) The learning unit 21 uses the spectrogram of the sound pressure signal and the vibration acceleration signal input from the signal memory unit 13 to learn the weights W of the neural network 94 that generates a noise removal mask α for removing noise components (noise A91b) from the input physical quantity, which is the electronic device vicinity sound A90, and the noise reduction input physical quantity A90A selected by the selection unit 16. (5) The learning unit 21 uses the spectrogram of the sound pressure signal and the vibration acceleration signal input from the signal memory unit 13 to learn a transfer function H that converts the extracted signal extracted by the generated noise removal mask α into an estimated vibration acceleration A92 of the electronic component 101. (6) The learning unit 21 extracts the extracted operation sound A91a as the extracted signal instead of the extracted knocking sound 91a (FIG. 3A). (7) The learning unit 21 learns the weights of the neural network so that the correlation between the first signal Sfi acquired by performing arbitrary processing on the noise A91b, which is a noise component, and the observed vibration acceleration A93, which is a teacher signal, becomes small, and the correlation between the second signal Sse acquired by performing arbitrary processing on the extracted operation sound A91a, which is an extracted signal, and the observed vibration acceleration A93, which is a teacher signal, becomes large. Specifically, the learning unit 21 learns the weights W and the transfer function H of the neural network that generates the noise removal mask α so that the coherence between the first signal Sfi acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise A91b and the observed vibration acceleration A93 becomes small, and the coherence between the second signal Sse acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted operation sound A91a and the observed vibration acceleration A93 becomes large. (8) Furthermore, the learning unit 21 learns the weights W and transfer function H of the neural network that generates the noise removal mask α using the neural network 94 so that the error between the spectrogram obtained by performing a short-time Fourier transform (STFT) on the observed vibration acceleration A93, which is the teacher signal, and the estimated vibration acceleration A92, which is the estimated signal, is minimized.

[0167] In such a modified application example, the signal processing device 10 treats an electronic device nearby sound A90 (FIG. 21), which is a sound at the ear position of a listener, as an input physical quantity, instead of an engine nearby sound 90 (FIG. 3A). Moreover, the signal processing device 10 treats an extracted operation sound A91a (FIG. 21), which is an operation sound of the electronic component 101 to be verified, as an extracted signal, instead of an extracted knocking sound 91a (FIG. 3A). Moreover, the signal processing device 10 treats a noise A91b (FIG. 21) as a noise component, instead of a noise 91b (FIG. 3A). Moreover, the signal processing device 10 treats an estimated vibration acceleration A92 (FIG. 21) of the electronic component 101 to be verified as an estimated signal, instead of an estimated knocking in-cylinder pressure 92 (FIG. 3A). Moreover, the signal processing device 10 treats an observed vibration acceleration A93 (FIG. 21) of the electronic component 101 to be verified as a teacher signal, instead of an observed knocking in-cylinder pressure 93 (FIG. 3A). Then, the signal processing device 10 executes a learning mode, a threshold calculation mode, a judgment mode, a separation mode, and a sensory test mode using these signals. In the sensory test mode, the signal processing device 10 can verify what kind of operation sound is generated by the electronic component 101.

[0168] The signal processing device 10Z according to the comparative example can also be used in a modified application example in which the verification target is one of a plurality of electronic components mounted on an electronic device, and the type of operation sound generated by the electronic component to be verified is verified. However, in the learning mode when the signal processing device 10Z according to the comparative example is used in this modified application example, if the noise components contained in the input physical quantity are relatively large, a missed signal SG13 (a signal that was not extracted) (FIG. 22C) may be generated when removing the noise components from the input physical quantity to extract an extracted signal, and the extracted signal may not be separated accurately. In other words, in the learning mode, the signal processing device 10Z according to the comparative example may not be able to neatly separate the signal to be extracted (the signal to be extracted SG12 (FIG. 22B)) from the input physical quantity when the noise components contained in the input physical quantity are relatively large.

[0169] 22A to 22D are explanatory diagrams of inappropriate examples (examples in which a signal SG11 (a signal to be extracted) cannot be neatly separated from an input physical quantity) that occur in the signal processing device 10Z according to the comparative example. Here, the description is given on the assumption that the signal processing device 10Z according to the comparative example is used in a change of use example to verify what kind of operating sound is generated by one of a plurality of electronic components mounted on an electronic device as a verification target.

[0170] 22A is an example of a spectrogram of an input physical quantity. The input physical quantity contains a signal SG11 to be taken (a signal to be extracted) and noise NO11.

[0171] FIG. 22B shows a spectrogram of an extracted signal obtained by extracting a signal SG12 (a signal to be extracted) to be taken from the input physical quantity shown in FIG. 22A.

[0172] Fig. 22C is a spectrogram of the noise components after the extraction signal is extracted from the input physical quantity shown in Fig. 22A. The noise components include a signal SG13 (a signal that was not extracted) that was missed when extracting the extraction signal and noise NO11. Since the missed signal SG13 has a lower sound pressure than the noise NO11, the coherence between the noise components and the teacher signal is low (Fig. 22D).

[0173] FIG. 22D shows the coherence between the noise component and the teacher signal in an inappropriate example. As shown in FIG. 22D, in the inappropriate example, the coherence between the noise component containing the missed signal SG13 and the teacher signal is low. In this case, in the learning of the weight W and the transfer function H of the neural network by the learning unit 21, learning stops even though the missed signal SG13 is included in the noise component, and the missed signal occurs. In this regard, if there is no noise NO11 (if the sound pressure of the noise is small) (FIG. 23C), the coherence between the noise component containing the missed signal SG13 and the teacher signal is high (FIG. 23D). The signal processing device 10 according to the first embodiment is improved so that the learning of the weight W and the transfer function H of the neural network does not stop by increasing the coherence between the noise component containing the missed signal SG13 and the teacher signal as shown in FIG. 23D.

[0174] Figures 23A to 23E are explanatory diagrams of a preferred example (an example where a desired signal SG21 (a signal to be extracted) can be neatly separated from an input physical quantity) realized in the signal processing device 10 of the first embodiment.

[0175] 23A is an example of a spectrogram of an input physical quantity in a preferred example. The input physical quantity contains a signal SG21 to be taken (a signal to be extracted) and noise NO21.

[0176] Fig. 23B shows a spectrogram of the noise-reduced input physical quantity including the signal SG22 to be removed (signal to be extracted) after the noise NO21 is reduced from the input physical quantity shown in Fig. 23A using a processing mask. The noise-reduced input physical quantity shown in Fig. 23B includes the signal SG22 to be removed and the noise NO22 reduced by the processing mask.

[0177] Fig. 23C is a spectrogram of noise components after an extraction signal is extracted from the noise-reduced input physical quantity shown in Fig. 23B during learning of the weights W and transfer functions H of the neural network by the learning unit 21. The noise components in the example shown in Fig. 23C include a missed signal SG23, and the sound pressure of the noise NO23 is smaller than that of the missed signal SG23. Therefore, the coherence between the noise components and the teacher signal after an extraction signal is extracted from the noise-reduced input physical quantity shown in Fig. 23B becomes high (Fig. 23D).

[0178] FIG. 23D shows the coherence between the noise component and the teacher signal in a suitable example. As shown in FIG. 23D, in a suitable example, the coherence between the noise component containing the missed signal SG23 and the teacher signal is high. The signal processing device 10 according to the present first embodiment can prevent learning from stopping when the missed signal SG23 is included in the noise component during learning of the weight W and the transfer function H of the neural network by increasing the coherence between the noise component containing the missed signal SG23 and the teacher signal as shown in FIG. 23D. In the inappropriate case shown in FIG. 22D, in learning of the weight W and the transfer function H of the neural network by the learning unit 21, even if the missed signal SG13 is included in the noise component, learning stops, and an error occurs. The signal processing device 10 according to the present first embodiment can suppress the error.

[0179] Fig. 23E is a spectrogram of noise components after an extraction signal is extracted from the spectrogram of the input physical quantity in the preferable example of Fig. 23A. The sound pressure of the missed signal SG24 in the preferable example shown in Fig. 23E is smaller than the sound pressure of the missed signal SG13 in the inappropriate example shown in Fig. 22C. In this way, the signal processing device 10 according to the first embodiment can reduce the sound pressure of the missed signal SG24.

[0180] In order to prevent the learning from stopping midway, the signal processing device 10 according to the first embodiment learns the weights W and the transfer function H of the neural network while generating a noise-reduced input physical quantity A90A, which is data obtained by reducing noise in a portion where noise is dominant from the electronic device vicinity sound A90, which is an input physical quantity. In the learning mode, if the sound pressure of the noise contained in the input physical quantity is relatively large, it becomes impossible to neatly separate the signal to be taken (signal to be extracted) from the input physical quantity. Therefore, in order to improve the separation performance of the signal to be taken (signal to be extracted), the signal processing device 10 according to the first embodiment is provided with the noise-reduced input physical quantity generation function and the shuffle function. The noise-reduced input physical quantity generation function is realized by the noise-reduced input physical quantity generation unit 15 (FIGS. 3A and 3B). The shuffle function is realized by the selection unit 16 (FIGS. 3A and 3B).

[0181] Fig. 24 is a schematic explanatory diagram of a method for generating a processing mask β. The processing mask β operates to pass a signal in a portion of the spectrogram of the electronic device vicinity sound A90 where noise is not dominant, while reducing noise in a portion where noise is dominant. The processing mask β is generated by filtering the noise removal mask α. In the example shown in Fig. 24, an example is shown in which the noise removal mask α is generated by filtering using a method that uses max pooling and upsampling.

[0182] FIG. 25 is an explanatory diagram of a method for generating a processing mask β. FIG. 25 shows an overview of a method using max pooling and upsampling. As shown in FIG. 25, the method using max pooling and upsampling replaces all elements in a specified area with the element of the maximum value. In addition, a maximum filter, a binarization method, or a combination of these may be used. This extends a portion where noise is not dominant by a specified area, preventing the signal to be extracted from being mistakenly reduced.

[0183] The noise-reduction input physical quantity generating unit 15 of the signal processing device 10 generates the processing mask β from the noise elimination mask α by using these methods. Note that these methods are merely examples, and the noise-reduction input physical quantity generating unit 15 can generate the processing mask β from the noise elimination mask α by using methods other than these.

[0184] 26 and 27 are explanatory diagrams of a method for generating the noise-reduced input physical quantity A90A, respectively. FIG. 26 and FIG. 27 show that the noise-reduced input physical quantity generation unit 15 generates the noise-reduced input physical quantity A90A by calculating an element product of the electronic-device vicinity sound A90 and a processing mask β. Such a noise-reduced input physical quantity generation unit 15 can generate a spectrogram of the noise-reduced input physical quantity A90A in which a portion in which noise is dominant is reduced from the electronic-device vicinity sound A90 using the processing mask β. The noise-reduced input physical quantity A90A generated by the noise-reduced input physical quantity generation unit 15 is supplied to the selection unit 16 as a data set associated with the observed vibration acceleration A93, which is a teacher signal.

[0185] Fig. 28 is an explanatory diagram of the shuffle process in the learning mode of the signal processing device 10 according to the first embodiment. Fig. 28 shows that a data set DS90 in which a plurality of electronic device vicinity sounds A90 are associated with the observed vibration accelerations A93 that are corresponding teacher signals is prepared, and a data set DS90A in which a plurality of noise reduction input physical quantities A90A are associated with the observed vibration accelerations A93 that correspond to the noise reduction input physical quantities is prepared. Fig. 28 shows that the selection unit 16 shuffles the data set DS90 and the data set DS90A and supplies them to the learning unit 21. Note that the observed vibration accelerations A93 of the data set DS90 and the observed vibration accelerations A93 of the data set DS90A are the same.

[0186] The learning unit 21 learns the weights W and the transfer function H of the neural network based on the data set selected by the selection unit 16 from the data set DS90 and the data set DS90A. At this time, the spectrogram of the noise-reduced input physical quantity A90A included in the data set DS90A has noise reduced in the noise-dominant portion by the processing mask β, so that the coherence between the signal SG22 to be taken and the teacher signal can be increased as shown in Fig. 23D. When learning the weights W and the transfer function H of the neural network, the learning unit 21 performs learning so that the coherence is reduced.

[0187] In the signal processing device 10 according to the first embodiment, even if the sound pressure of the noise is relatively larger than the signal to be taken (signal to be extracted) from the input physical quantity, the missed signal is included in the noise component, but the degree of association (e.g., coherence) between the noise component and the teacher signal is reduced, and learning can be prevented from stopping. Therefore, the signal processing device 10 according to the present embodiment can suppress the occurrence of missed signals in learning of the weights W and transfer functions H of the neural network by the learning unit 21.

[0188] Incidentally, as shown in FIG. 29, the learning unit 21 of the signal processing device 10 according to the present first embodiment can be configured to learn the weights W of the neural network without using the transfer function H during the learning mode. That is, the learning unit 21 of the signal processing device 10 according to the present first embodiment can be configured not to optimize part of the learning during the learning mode. FIG. 29 is an explanatory diagram of a modified example of the operation of the signal processing device 10 according to the present first embodiment during the learning mode. As shown in FIG. 29, the learning unit 21 of the modified example is different from the learning unit 21 of the example shown in FIG. 3A in that a determination process (determination process for minimizing error) that reduces the correlation between the estimated vibration acceleration A92, which is an estimated signal, and the observed vibration acceleration A93, which is a teacher signal, is deleted. Therefore, the learning unit 21 of the modified example is configured, during learning mode, to only perform a determination process (determination process for minimizing coherence) that reduces the correlation between the noise component A91b and the teacher signal, the observed vibration acceleration A93, and a determination process (determination process for maximizing coherence) that increases the correlation between the extracted operation sound A91a, which is the extracted signal, and the teacher signal, the observed vibration acceleration A93.

[0189] <Main features of the signal processing device (estimation device)> The signal processing device 10 according to the present embodiment mainly has the following characteristics. In the following description, the "input physical quantity" means, for example, the "engine vicinity sound 90" in the application example shown in FIG. 3A, and means the "electronic device vicinity sound A90" in the application change example shown in FIG. 21. The "noise reduction input physical quantity" means, for example, the "noise reduction input physical quantity 90A" in the application example shown in FIG. 3A, and means the "noise reduction input physical quantity A90A" in the application change example shown in FIG. 21. The "extracted signal" means, for example, the "extracted knocking sound 91a" in the application example shown in FIG. 3A, and means the "extracted operation sound A91a" in the application change example shown in FIG. The "noise component" means, for example, the "noise 91b" in the application example shown in FIG. 3A, and means the "noise A91b" in the application change example shown in FIG. The "estimated signal" means, for example, the "estimated knocking cylinder pressure 92" in the application change example shown in FIG. 21, and means the "estimated vibration acceleration A92". Moreover, the "teacher signal" means, for example, the "observed knocking in-cylinder pressure 93" in the application example shown in FIG. 3A, and means the "observed vibration acceleration A93" in the application change example shown in FIG.

[0190] (1) As shown in Fig. 2, the signal processing device 10 according to the present embodiment includes a noise-reduced input physical quantity generating unit 15, a selecting unit 16, and a learning unit 21. As shown in Fig. 3A and Fig. 21, the noise-reduced input physical quantity generating unit 15 generates a noise-reduced input physical quantity in which a noise component included in an input physical quantity is reduced. That is, in the signal processing device 10 according to the present embodiment, the noise-reduced input physical quantity generating unit 15 generates a noise-reduced input physical quantity as data with low noise. The selecting unit 16 selects either or both of the input physical quantity and the noise-reduced input physical quantity, and supplies them to the learning unit 21. The learning unit 21 learns weights W of a neural network that generates a noise removal mask α for removing noise components from the input physical quantity and the noise-reduced input physical quantity selected by the selecting unit 16.

[0191] Here, "the input physical quantity and the noise-reduced input physical quantity selected by the selection unit 16" specifically means a dataset selected by the selection unit 16 from dataset DS90 (FIG. 28) in which a plurality of electronic-device vicinity sounds A90 (input physical quantities) are associated with the corresponding observed vibration accelerations A93 (teacher signals) and dataset DS90A (FIG. 28) in which a plurality of noise-reduced input physical quantities A90A are associated with the corresponding observed vibration accelerations A93 (teacher signals).

[0192] The signal processing device 10 according to this embodiment can realize a signal processing method including: a noise-reduced input physical quantity generation step of generating a noise-reduced input physical quantity in which a noise component contained in an input physical quantity has been reduced; a selection step of selecting either the input physical quantity or the noise-reduced input physical quantity, or both; and a learning step of learning weights of a neural network that generates a noise removal mask for removing the noise component from the input physical quantity and the noise-reduced input physical quantity selected in the selection step.

[0193] The signal processing device 10 according to this embodiment can prevent learning from stopping due to a decrease in the degree of association (e.g., coherence) between the noise components and the teacher signal, even when the amount of noise is large (particularly when the sound pressure of the noise is relatively larger than the signal to be taken from the input physical quantity (signal to be extracted)), despite the missed signal being included in the noise components. Therefore, the signal processing device 10 according to this embodiment can suppress the occurrence of missed signals in learning of the weights W and transfer function H of the neural network by the learning unit 21. As a result, the signal processing device 10 according to this embodiment can perform learning to satisfactorily separate the input physical quantity into the noise components and the extracted signal, even when the amount of noise is large.

[0194] (2) As shown in FIGS. 3A and 21 , the noise-reduced input physical quantity generation unit 15 of the signal processing device 10 according to this embodiment may generate data in which noise components have been reduced from the input physical quantity (noise-reduced input physical quantity) by using a processing mask β obtained by filtering the noise removal mask α.

[0195] The signal processing device 10 according to this embodiment increases the coherence between the noise components containing the missed signals and the teacher signal, and when learning the weights W of the neural network, performs learning so as to reduce the coherence, thereby enabling good learning of the weights W of the neural network.

[0196] (3) As shown in FIG. 3A and FIG. 21, the learning unit 21 of the signal processing device 10 according to this embodiment may learn the weights W of the neural network so that the correlation between the noise components and the teacher signal is reduced and the correlation between the teacher signal and an extracted signal obtained by removing the noise components from the input physical quantity or the noise-reduced input physical quantity is increased.

[0197] The signal processing device 10 according to this embodiment can evaluate whether the noise components after separation contain components related to the teacher signal. Therefore, the signal processing device 10 according to this embodiment can perform learning for satisfactorily separating the input physical quantity into the noise components and the extracted signal by generating a noise-reduced input physical quantity in which the noise amount included in the input physical quantity (input signal) is reduced in the noise-reduced input physical quantity generating unit 15 and evaluating whether the noise components after separation contain components related to the teacher signal in the learning unit 21.

[0198] 3A and 21, the learning unit 21 of the signal processing device 10 according to this embodiment may use a noise removal mask α to generate an extracted signal by removing noise components from the input physical quantity or the noise-reduced input physical quantity, and may learn a transfer function H for converting the extracted signal into an estimated signal of the same dimension as the teacher signal. At that time, the learning unit 21 may learn the weight W and the transfer function H of the neural network so that the error of the estimated signal with respect to the teacher signal is small.

[0199] The signal processing device 10 according to this embodiment can learn optimal neural network weights W and transfer function H by learning the neural network weights W and transfer function H so that the error of the estimated signal relative to the teacher signal is small.

[0200] (5) As shown in Fig. 3A and Fig. 21, the learning unit 21 of the signal processing device 10 according to the present embodiment learns the weights W of the neural network that generates a noise removal mask α for removing noise components by adding a phase from the input physical quantity and the noise-reduced input physical quantity selected by the selection unit 16. The learning unit 21 also learns a transfer function H for converting an extracted signal from which a noise component has been removed from the input physical quantity and the noise-reduced input physical quantity selected by the selection unit 16 into an estimated signal of the same dimension (unit) as the teacher signal by adding a phase. In this configuration, as shown in Fig. 3A and Fig. 21, the learning unit 21 learns the weights W of the neural network 94 so that the correlation between the noise component and the teacher signal is reduced and the correlation between the extracted signal and the teacher signal is increased.

[0201] The signal processing device 10 according to the present embodiment generates a noise-reduced input physical quantity in which the amount of noise contained in the input physical quantity (input signal) is reduced when learning the weight W of the neural network 94, and evaluates whether the noise component after separation contains a component related to the teacher signal, thereby performing learning for separating the input physical quantity into a noise component and an extracted signal in a satisfactory manner. Such a signal processing device 10 can satisfactorily separate the noise component from a signal suitable for level change in a sensory test. Furthermore, the signal processing device 10 can easily evaluate whether a target sound (knocking sound or an operating sound of an electronic component to be verified) is mixed into a background sound. Therefore, the signal processing device 10 can improve the evaluation performance of a knocking sound or an operating sound of an electronic component to be verified, as compared with the conventional techniques described in, for example, Patent Documents 2 and 3. Furthermore, the signal processing device 10 can perform a satisfactory sensory test, and can make a judgment close to that of an inspector by determining a threshold value based on data that is unacceptable in the sensory test.

[0202] (6) As shown in FIGS. 3A and 21, the learning unit 21 of the signal processing device 10 according to this embodiment learns the weights W and the transfer function H of the neural network that generates the noise removal mask α, taking into account the amplitude and phase components related to the input physical quantity.

[0203] The signal processing device 10 according to the present embodiment can separate the input physical quantity into an extracted signal and a noise component so that a knocking sound or an operating sound of an electronic component to be verified (target sound) is not mixed into the noise. The signal processing device 10 can improve the evaluation performance of a knocking sound or an operating sound of an electronic component to be verified compared to the conventional techniques described in Patent Documents 2 and 3. Moreover, the signal processing device 10 can perform a good sensory test.

[0204] (7) The learning unit 21 of the signal processing device 10 according to the present embodiment learns the weights W and the transfer function H of the neural network so that the correlation between the noise components and the teacher signal is reduced and the correlation between the extracted signal and the teacher signal is increased. In this case, for example, as shown in Fig. 3A and Fig. 21, the learning unit 21 of the signal processing device 10 according to the present embodiment may use the coherence between the noise components and the teacher signal as an element representing the correlation between the noise components and the teacher signal. In addition, the learning unit 21 of the signal processing device 10 according to the present embodiment may use the coherence between the extracted signal and the teacher signal as an element representing the correlation between the extracted signal and the teacher signal. Specifically, as shown in Figures 3A and 21, the learning unit 21 of the signal processing device 10 according to this embodiment learns the weights W and transfer function H of the neural network so that the coherence (or coherent output power) between the signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on the noise component and the teacher signal is reduced, and the coherence (or coherent output power) between the signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on the extracted signal is increased.

[0205] The signal processing device 10 according to this embodiment can remove sounds caused by the knocking in-cylinder pressure and the operation of the electronic component to be verified from the noise components.

[0206] (8) As shown in Fig. 3A and Fig. 21, the learning unit 21 of the signal processing device 10 according to the present embodiment learns the weights W and the transfer function H of the neural network so that the error of the estimated signal with respect to the teacher signal is reduced. Specifically, the learning unit 21 of the signal processing device 10 learns so that the error of the estimated signal with respect to a spectrogram obtained by performing a short-time Fourier transform (STFT) on the teacher signal is reduced. Alternatively, the learning unit 21 of the signal processing device 10 learns so that the error with respect to a signal waveform of an estimated signal obtained by performing an inverse short-time Fourier transform (ISTFT) and a fast Fourier transform (FFT) on the teacher signal and the extracted signal, multiplying the signal by the transfer function H, and performing an inverse fast Fourier transform (IFFT) is reduced.

[0207] The signal processing device 10 according to the present embodiment can learn the weights W and the transfer function H of the neural network so that the error of the estimated signal with respect to the spectrogram obtained by performing a short-time Fourier transform (STFT) on the teacher signal is reduced. Alternatively, the signal processing device 10 can learn the weights W and the transfer function H of the neural network so that the error of the estimated signal with respect to the signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) and a fast Fourier transform (FFT) on the teacher signal and the extracted signal, multiplying the transfer function H, and performing an inverse fast Fourier transform (IFFT) is reduced.

[0208] (9) As shown in FIG. 7B, the signal processing device 10 according to this embodiment includes a signal synthesis unit 50 that changes the level of an extracted signal and synthesizes it with a noise component separated from the input physical quantity to generate a processed sound.

[0209] The signal processing device 10 according to this embodiment can make the target sound to be inspected (knocking sound / operation sound of the electronic component to be verified) easily audible. The signal processing device 10 can allow an inspector to grasp the presence or absence of the target sound (knocking sound / operation sound of the electronic component to be verified) with high accuracy, and can determine a threshold value based on data that is unacceptable in the sensory test, thereby enabling a judgment close to that of an inspector.

[0210] (10) As shown in FIG. 7B, the signal synthesis unit 50 of the signal processing device 10 according to this embodiment has a signal adjustment unit 51 that accepts a designation of the level of an extracted signal by an examiner or an operator.

[0211] The signal processing device 10 according to this embodiment can arbitrarily and minutely change the level of the extracted signal.

[0212] (11) As shown in Fig. 14A and Fig. 17, the signal processing device 10 according to the present embodiment can realize the following signal processing method. That is, the signal processing method according to the present embodiment includes a learning step (steps S30 to S32 in Fig. 14A) and a separation step (steps S60 to S62 in Fig. 17). In the learning step, a weight W of a neural network 94 that generates a noise removal mask α for removing a noise component from an input physical quantity containing a noise component is learned. At this time, the weight W of the neural network 94 is learned so that the correlation between the noise component and the teacher signal is reduced and the correlation between the extracted signal and the teacher signal is increased. At this time, the neural network 94 may learn the weight W of the neural network 94 that generates the noise removal mask α for removing the noise component from the input physical quantity containing the noise component while taking into account the phase, and the transfer function H for converting the extracted signal obtained by removing the noise component from the input physical quantity into an estimated signal of the same dimension (unit) as the teacher signal while taking into account the phase. In the separation step, the noise removal mask α is used to separate the input physical quantity into a noise component and an extracted signal from which the noise component has been removed.

[0213] The signal processing method according to the present embodiment can effectively separate an input physical quantity into a noise component and an extracted signal, particularly into a noise component and a signal suitable for level change in a sensory test.

[0214] 3A and 21, the signal processing device 10 according to this embodiment is an estimation device that estimates an estimated signal of the same dimension (unit) as the teacher signal by multiplying an extracted signal obtained by removing noise components from an input physical quantity by a transfer function H. The learning unit 21 of the signal processing device 10 according to this embodiment learns, by a neural network 94, the weights W and the transfer function H of a neural network that generates a noise removal mask α for removing noise components from the input physical quantity and extracting an extracted signal.

[0215] The signal processing device 10 according to this embodiment can learn the weights W and the transfer function H of the neural network by taking into account the amplitude and phase components related to the input physical quantity with respect to the noise removal mask α and the transfer function H.

[0216] (13) The signal processing device 10 according to the present embodiment can realize the following estimation method. That is, the estimation method according to the present embodiment is a method of estimating an estimated signal of the same dimension (unit) as the teacher signal by multiplying an extracted signal obtained by removing noise components from an input physical quantity by a transfer function H. As shown in FIG. 14B or FIG. 14C, the estimation method according to the present embodiment includes a learning step (step S103 or step S103a) and an estimated signal estimation step (step S102). In the learning step (step S103 or step S103a), the neural network 94 learns a weight W and a transfer function H of a neural network that generates a noise removal mask α for removing noise components from the input physical quantity and extracting an extracted signal. In the estimated signal estimation step (step S102), the neural network 94 uses the noise removal mask α to obtain an extracted signal from which noise components have been removed, and converts the extracted signal into an estimated signal by multiplying the extracted signal by the transfer function H. In the estimation method according to the present embodiment, in the learning step (step S103 or step S103a), when learning the weight W and the transfer function H of the neural network, the noise removal mask α and the transfer function H are learned while taking into account the amplitude and phase components related to the input physical quantity.

[0217] In the estimation method according to this embodiment, when learning the weight W and the transfer function H of the neural network, the noise removal mask α and the transfer function H can be learned while taking into account the amplitude and phase components related to the input physical quantity.

[0218] As described above, according to the signal processing device 10 of this embodiment, even when the amount of noise is large (particularly when the sound pressure of the noise is relatively larger than the signal to be taken from the input physical quantity (signal to be extracted)), it is possible to avoid a situation in which the coherence between the noise component and the teacher signal decreases and learning stops despite the missed signal being included in the noise component. Therefore, the signal processing device 10 of this embodiment can suppress the occurrence of missed signals in learning of the weight W and transfer function H of the neural network by the learning unit 21. As a result, the signal processing device 10 of this embodiment can perform learning to satisfactorily separate the input physical quantity into the noise component and the extracted signal even when the amount of noise is large.

[0219] Moreover, according to the signal processing device 10 of the first embodiment, when learning the weight W of the neural network 94, a noise-reduced input physical quantity in which the noise amount contained in the input physical quantity (input signal) is reduced is generated, and while evaluating whether the noise component after separation contains a component related to the teacher signal, the evaluation result is added to the learning data, thereby making it possible to perform learning for separating the input physical quantity into a noise component and an extraction signal in a good manner. Moreover, according to the signal processing device 10 of the first embodiment, the input physical quantity can be separated into a noise component and an extraction signal in a good manner. In particular, the signal processing device 10 can separate the noise component into a signal suitable for level change in a sensory test. Such a signal processing device 10 can make it easy to evaluate whether or not a target sound (knocking sound / operation sound of an electronic component to be verified) is mixed into the background sound. Therefore, the signal processing device 10 can improve the evaluation performance of the knocking sound and the operation sound of an electronic component to be verified. Moreover, the signal processing device 10 can perform a good sensory test.

[0220] [Second embodiment] The configuration of a signal processing device 10A (estimation device) according to the second embodiment will be described with reference to Fig. 30. Fig. 30 is a block diagram showing the configuration of the signal processing device 10A (estimation device) according to the second embodiment.

[0221] The signal processing system 100A in Fig. 30 is realized by a signal processing device 10A. As shown in Fig. 30, the signal processing device 10A (estimation device) according to the second embodiment is different from the signal processing device 10 according to the first embodiment (see Fig. 2) in that the signal synthesis unit 50 has the following function. That is, the signal synthesis unit 50 has a function of multiplying the teacher signal by the inverse of the transfer function H, and synthesizing it with the noise component separated from the input physical quantity and the noise-reduced input physical quantity selected by the selection unit 16 to generate a processed sound.

[0222] The signal processing device 10A (estimation device) according to this second embodiment, like the signal processing device 10 according to the first embodiment, can enable an inspector to grasp with high accuracy the presence or absence of a target sound (such as the knocking sound of the engine 1 or the operating sound of the electronic component 101), thereby improving inspection performance.

[0223] The signal synthesis unit 50 may have a function of multiplying the level-changed teacher signal, which is obtained by changing the level (magnitude) of the teacher signal, by the inverse of the transfer function H, and synthesizing the multiplied signal with the noise component separated from the input physical quantity and the noise-reduced input physical quantity selected by the selection unit 16 to generate a processed sound.

[0224] In addition, the signal synthesis unit 50 may have a function of multiplying an arbitrary signal (e.g., an arbitrary knocking in-cylinder pressure signal or a vibration acceleration signal) having the same dimension as the teacher signal, instead of the teacher signal, by the inverse of the transfer function H, and synthesizing the multiplied signal with the noise component separated from the input physical quantity and the noise reduction input physical quantity selected by the selection unit 16 to generate a processed sound.

[0225] In addition, the signal synthesis unit 50 may multiply the teacher signal by the inverse of the modified transfer function Hc obtained by modifying the value of the transfer function H, and synthesize the multiplied signal with the noise component separated from the input physical quantity and the noise-reduced input physical quantity selected by the selection unit 16 to generate a processed sound.

[0226] In addition, the signal synthesis unit 50 may have a function of multiplying the level-changed teacher signal, which is obtained by changing the level (magnitude) of the teacher signal, by the inverse of a changed transfer function Hc, which is obtained by changing the value of the transfer function H, and synthesizing the multiplied signal with a noise component separated from the input physical quantity and the noise-reduced input physical quantity selected by the selection unit 16 to generate a processed sound.

[0227] Here, for example, the modified transfer function Hc is a function in which the amplitude corresponding to a certain frequency band of the transfer function H is increased or decreased and / or the phase is changed.

[0228] The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the gist of the present invention.

[0229] For example, the above-described embodiment has been described in detail to clearly explain the gist of the present invention. Therefore, the present invention is not necessarily limited to having all of the components described. In addition, the present invention can be modified by adding other components to certain components, or by replacing some components with other components. In addition, the present invention can be modified by deleting some components. [Explanation of symbols]

[0230] 1. Engine 2. Engine ECU 3. Vehicle 4...Sound pressure sensor 5...Cylinder pressure sensor 6. Data collection device 7. Monitor 8...Headphones (sound output section) 9…Level designation section 10, 10A, 10Z...Signal processing device (estimation device) 11...Signal cutting section 12...Spectrogram calculation section 13...Signal storage section 14…Switch 15…Noise reduction input physical quantity generator 16...Selection section 20, 20Z…Learning processing section 21,21Z…Study section 22…Learned parameter memory section 23…1st estimation part 24...Threshold calculation unit 25...Threshold memory unit 26a...Teacher signal storage section 26b... Estimated signal storage unit 26c…Noise component storage section 26d...extracted signal storage section 30...Determination processing unit 31...Second estimation unit (extracted signal estimation unit) 32...Threshold value judgment unit 40…Separation part 50...Signal synthesis section 51...Signal conditioning section 52...Signal output section 81...Knocking vibration 82...Knocking sound 83...Mechanical noise (noise component) 90…Near-engine noise (input physical quantity) 90A…Noise reduction input physical quantity 91a…Extracted knocking sound (extracted signal) 91aa…Level change knocking sound (level change extraction signal) 91b…Noise (noise component) 91c…processed sound 92…Estimated knocking cylinder pressure (estimated signal) 93…Observed knocking cylinder pressure (teacher signal) 94…Neural network 94A…Mask generation network 95…U-Net 96…Downward pass (Encoder) 97…Hierarchy 98…Upward path (Decoder) 99…Output 100, 100A, 100Z...Signal processing system 101...Electronic components (verification subject) 102...Electronic components 103...Electronic equipment 105...Accelerometer α…Noise reduction mask β…Processing mask A90…Near electronic device noise (input physical quantity) A90A…Noise reduction input physical quantity A91a…Extraction operation sound (extraction signal) A91b…Noise (noise component) A92…Estimated vibration acceleration (estimated signal) A93…Observed vibration acceleration (teacher signal) DS90, DS90A... Data set F11, F21, F31, F37...Spectrogram F12,F22,F32,F36,F93…Signal waveform F13, F23…Coherence F33, F35...Spectrum F34…Frequency response characteristics H,He…Transfer function Hc…changed transfer function M1…Connection for learning mode M2: Connection for threshold calculation mode M3: Connection for judgment mode M4…Connection for separation mode M5: Connection for sensory test mode NO11, NO21, NO22...Noise SG11, SG12, SG21, SG22...Signal to be taken (signal to be extracted) SG13, SG23: Missed signals (signals that were not extracted) W…weight T: Threshold

Claims

1. a noise-reduced input physical quantity generating unit for generating a noise-reduced input physical quantity by reducing a noise component included in the input physical quantity; a selection unit that selects one or both of the input physical quantity and the noise-reduction input physical quantity; a learning unit configured to learn weights of a neural network that generates a noise reduction mask for removing noise components from the input physical quantity and the noise reduction input physical quantity selected by the selection unit, the noise-reduced input physical quantity generation unit generates the noise-reduced input physical quantity by reducing the noise component from the input physical quantity using a processing mask obtained by filtering the noise removal mask; the selection unit supplies a data set obtained by shuffling the input physical quantity and the noise-reduced input physical quantity to the learning unit; The learning unit learns weights of the neural network so that a correlation between the noise component and a teacher signal becomes small and a correlation between the teacher signal and an extracted signal obtained by removing the noise component from the input physical quantity or the noise-reduced input physical quantity becomes large.

23. A signal processing device comprising:

2. 2. The signal processing device according to claim 1, The learning unit uses the noise removal mask to generate the extracted signal by removing the noise component from the input physical quantity or the noise-reduced input physical quantity, and learns a transfer function for converting the extracted signal into an estimated signal of the same dimension as the teacher signal.

23. A signal processing device comprising:

3. 3. The signal processing device according to claim 2, The learning unit learns the weights of the neural network and the transfer function so that an error of the estimated signal with respect to the teacher signal is reduced.

23. A signal processing device comprising:

4. a noise-reduced input physical quantity generating step of generating a noise-reduced input physical quantity by reducing a noise component included in the input physical quantity; a selection step of selecting one or both of the input physical quantity and the noise-reduction input physical quantity; a learning step of learning weights of a neural network that generates a noise reduction mask for removing noise components from the input physical quantity and the noise reduction input physical quantity selected in the selection step, the noise-reduced input physical quantity generating step generates the noise-reduced input physical quantity by reducing the noise component from the input physical quantity using a processing mask obtained by filtering the noise removal mask; In the selection step, a data set obtained by shuffling the input physical quantity and the noise-reduced input physical quantity is supplied to a learning unit that executes processing in the learning step; In the learning step, weights of the neural network are learned so that the correlation between the noise component and a teacher signal becomes small and the correlation between the teacher signal and an extracted signal obtained by removing the noise component from the input physical quantity or the noise-reduced input physical quantity becomes large. A signal processing method comprising:

Images