Signal processing device and signal processing method

Abstract

To reduce a learning time and properly separate noise components and extraction signals from target sound for which a desired clean signal cannot be prepared in advance.SOLUTION: A signal processing apparatus 10 includes: a learning unit 21 which learns a weight W of a neural network (mask generation network 94A) for generating a mask α for removing a noise component (noise 91b) from an input physical quantity (engine peripheral sound 90); and a separation unit 40 which separates, using the mask, the input physical quantity into the noise component and an extraction signal (extraction knocking sound 91a). The learning unit learns a weight of the neural network so that a relation between the noise component and a training signal (observation knocking cylinder internal pressure 93) may be reduced while a relation between the extraction signal and the training signal may be increased.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a signal processing device and a signal processing method that satisfactorily separates an input physical quantity into a noise component and an extracted signal from which the noise component has been removed. [Background technology]

[0002] For example, in an internal combustion engine such as a gasoline engine, the ignition timing is generally advanced as much as possible within a crank angle range where knocking does not occur, in order to improve output torque. Therefore, in the process of adjusting the ignition timing, a tester or a knocking determination device determines whether knocking occurs. An example of such a knocking determination device is described in Patent Document 1.

[0003] The knocking determination device described in Patent Document 1 selects a target signal to be compared with a determination signal for determining the presence or absence of knocking based on a condition determined by a relationship with the determination signal (for example, a relationship in terms of time or a relationship in terms of operating conditions).

[0004] In the knocking determination device described in Patent Document 1, only the determination result of the presence or absence of knocking can be presented to the outside, and it is difficult to verify the determination result. Therefore, the inventors of the present invention proposed the devices described in Patent Document 2 or Patent Document 3 as devices capable of estimating a desired physical quantity that can verify the determination result.

[0005] The device described in Patent Document 2 is described as "a learning device for removing noise components contained in an input physical quantity and estimating a desired physical quantity from the input physical quantity from which the noise components have been removed, wherein the noise components are noise other than a knocking sound generated in an internal combustion engine, the input physical quantity is the sound pressure of the internal combustion engine including the noise and the knocking sound, and the desired physical quantity is the in-cylinder pressure of the internal combustion engine when knocking occurs, and the learning device is characterized in that it includes a learning unit that removes the noise using a neural network and learns neural network weights that generate a mask that extracts the knocking sound, and / or a transfer function that converts the knocking sound extracted by the mask into the in-cylinder pressure of the internal combustion engine when knocking occurs."

[0006] The device described in Patent Document 3 is a "knocking determination device that removes noise other than the knocking sound generated in an internal combustion engine and estimates the knocking sound from which the noise has been removed, characterized in that the device includes: a learning unit that uses a neural network to learn network weights that remove the noise and generate a mask that extracts the knocking sound, and a transfer function that converts the knocking sound extracted by the mask into the in-cylinder pressure of the internal combustion engine at the time when knocking occurs; and a second estimation unit that uses the mask by a neural network to estimate the knocking sound from which the noise has been removed from the knocking sound including the noise."

[0007] Incidentally, another device using a neural network is described in Patent Document 4. The device described in Patent Document 4 is described as "a signal generating device including: a conversion unit that converts an acoustic signal and outputs amplitudes and phases at multiple frequencies; a frequency acquisition unit that acquires a frequency for each of multiple nodes in an intermediate layer included in a neural network that receives as input the amplitudes and phases, based on multiple weights used in the calculation of the node; and a generation unit that generates an acoustic signal based on the acquired multiple frequencies and the amplitudes and phases corresponding to each of the multiple nodes." [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2017-44148 A [Patent Document 2] Patent No. 6605170 [Patent Document 3] Patent No. 6651040 [Patent Document 4] Patent No. 6903611 Summary of the Invention [Problem to be solved by the invention]

[0009] However, the conventional techniques described in Patent Documents 2 and 3 have a problem in that it takes a long time to learn the transfer function portion (weighting). Furthermore, the conventional technology described in Patent Document 4 had a problem in that it was difficult to satisfactorily separate noise components and an extracted signal from a target sound (input physical quantity) where it is difficult to prepare a desired clean signal in advance, such as in a noisy environment with a large amount of machine sound and vibration.

[0010] The present invention has been made to solve the above-mentioned problems, and has as its main object to provide a signal processing device and a signal processing method that eliminates the need for learning the transfer function portion, thereby shortening the learning time, and satisfactorily separates noise components and an extracted signal from a target sound (input physical quantity) for which it is difficult to prepare a desired clean signal in advance. [Means for solving the problem]

[0011] In order to solve the above problems, the present invention provides Eliminates the need to learn a transfer function that converts the knocking sound extracted by removing noise components from the input physical quantity using a mask into the in-cylinder pressure of the internal combustion engine when knocking occurs. A signal processing device comprising: a learning unit configured to learn weights of a neural network that generates a mask for removing a noise component from an input physical quantity containing the noise component; and a separation unit configured to separate the input physical quantity into the noise component and an extracted signal from which the noise component has been removed, using the mask. The learning unit is configured to learn weights of a neural network that generates a mask for removing the noise component from an input physical quantity containing the noise component. The first signal obtained by performing an inverse short-time Fourier transform on the spectrogram of The correlation between the extracted signal and the teacher signal becomes smaller, and The second signal obtained by performing an inverse short-time Fourier transform on the spectrogram of The weights of the neural network are learned so that the correlation between the training signal and the teacher signal becomes large.

[0012] The present invention also provides a method for producing a method for manufacturing a semiconductor device comprising the steps of: This eliminates the need to learn a transfer function that converts the knocking sound extracted by removing noise components from the input physical quantity using a mask into the cylinder pressure of the internal combustion engine when knocking occurs. A signal processing method comprising: a learning step of learning weights of a neural network that generates a mask for removing a noise component from an input physical quantity containing the noise component; and a separation step of separating the input physical quantity into the noise component and an extracted signal from which the noise component has been removed, using the mask. In the learning step, The first signal obtained by performing an inverse short-time Fourier transform on the spectrogram of The correlation between the extracted signal and the teacher signal becomes smaller, and The second signal obtained by performing an inverse short-time Fourier transform on the spectrogram of The weights of the neural network are learned so that the correlation between the training signal and the teacher signal becomes large. Other means will be described later. Effect of the Invention

[0013] According to the present invention, by using the correlation between the noise component and the teacher signal and the correlation between the extracted signal and the teacher signal for learning, it is possible to eliminate the need to learn the transfer function portion and thereby shorten the learning time. In addition, it is possible to satisfactorily separate the noise component and the extracted signal from the target sound (input physical quantity) for which it is difficult to prepare a desired clean signal in advance. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an overall configuration of a signal processing system including a signal processing device according to an embodiment. [Diagram 2] 1 is a block diagram showing a configuration of a signal processing device according to an embodiment. [Figure 3A] FIG. 13 is an explanatory diagram of a learning mode. [Figure 3B] FIG. 4 is an explanatory diagram of an operation of the signal processing device according to the embodiment in a learning mode. [Figure 4A] FIG. 11 is an explanatory diagram of a threshold calculation mode. [Figure 4B] 1 is a diagram illustrating an operation of the signal processing device according to the embodiment in a threshold calculation mode. FIG. [Figure 5A] FIG. 11 is an explanatory diagram of a determination mode. [Figure 5B] FIG. 4 is an explanatory diagram of the operation of the signal processing device according to the embodiment in a determination mode. [Figure 6A] FIG. 13 is an explanatory diagram of a separation mode. [Figure 6B] FIG. 4 is an explanatory diagram of the operation of the signal processing device according to the embodiment in a separation mode. [Figure 7A] FIG. 13 is an explanatory diagram of a sensory test mode. [Figure 7B] 5A and 5B are explanatory diagrams illustrating the operation of the signal processing device according to the embodiment in a sensory test mode. [Figure 8A] FIG. 4 is an explanatory diagram illustrating the relationship between a noise component and a teacher signal during learning. [Figure 8B] FIG. 4 is an explanatory diagram illustrating the relationship between an extracted signal and a teacher signal during learning. [Figure 9] FIG. 10 is an explanatory diagram of learning weights of a neural network that generates a mask in an embodiment. [Figure 10A] FIG. 11 is an explanatory diagram of a calculation example in which phase components of pressure, vibration, sound, etc. are not taken into consideration. [Figure 10B] FIG. 11 is an explanatory diagram of a calculation example in which phase components of pressure, vibration, sound, etc. are taken into consideration. [Figure 11A] FIG. 1 is an explanatory diagram of signal processing in a sensory test (1). [Figure 11B] FIG. 2 is an explanatory diagram (2) of signal processing in a sensory test. [Figure 11C] FIG. 3 is an explanatory diagram of signal processing in a sensory test. [Figure 12A] FIG. 4 is an explanatory diagram of signal processing in a sensory test. [Figure 12B] FIG. 5 is an explanatory diagram (5) of signal processing in a sensory test. [Figure 12C] FIG. 6 is an explanatory diagram of signal processing in a sensory test. [Figure 13] 4 is a flowchart showing a data collection process of the signal processing device in the embodiment. [Figure 14A] 4 is a flowchart showing a learning process of the signal processing device in the embodiment. [Figure 14B] 13 is a flowchart showing a subroutine of a learning process. [Figure 15A] 5 is a flowchart showing a threshold calculation process of the signal processing device in the embodiment. [Figure 15B] 19 is a flowchart showing a threshold calculation process of the signal processing device that is performed after the separation process of FIG. 17 and the sensory test process of FIG. 18 in the embodiment. [Figure 16] 5 is a flowchart showing a determination process of the signal processing device in the embodiment. [Figure 17] 5 is a flowchart showing a separation process of the signal processing device in the embodiment. [Figure 18] 5 is a flowchart showing a sensory test process of the signal processing device in the embodiment. [Figure 19] FIG. 11 is an explanatory diagram of a first modified example. [Figure 20] FIG. 11 is an explanatory diagram of learning of weights of a neural network that generates a mask in the first modified example. [Figure 21] FIG. 11 is an explanatory diagram of a second modified example. [Figure 22] FIG. 11 is an explanatory diagram of learning of weights of a neural network that generates a mask in the second modified example. [Figure 23] FIG. 13 is an explanatory diagram of learning of weights of a neural network that generates a mask in the third modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] An embodiment of the present invention (hereinafter, referred to as "the present embodiment") will be described in detail with reference to the drawings. Note that each drawing is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to only the illustrated examples. In addition, in each drawing, common or similar components are given the same reference numerals, and duplicated explanations thereof will be omitted.

[0016] The conventional techniques described in Patent Documents 2 and 3 are configured to separate an input physical quantity into a noise component and an extracted signal without considering a phase component related to the input physical quantity. Therefore, even if the conventional techniques separate an input physical quantity into a noise component and an extracted signal, the extracted signal is mixed into the noise component, and the extracted signal cannot be accurately separated. Therefore, the present invention also intends to provide a signal processing device that satisfactorily separates an input physical quantity into a noise component and an extracted signal by realizing a configuration that considers the phase component.

[0017] In addition, in the conventional techniques described in Patent Documents 2 and 3, it is possible to select whether or not to estimate a desired physical quantity by removing the transfer function at the time of estimation after learning is completed. In contrast, the present invention also intends to provide a signal processing device that does not use the transfer function described in Patent Documents 2 and 3 even at the time of learning.

[0018] <Overall configuration of a signal processing system including a signal processing device> The overall configuration of a signal processing system 100 including a signal processing device 10 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the signal processing system 100 including the signal processing device 10.

[0019] The signal processing device 10 is a device that performs various processes on signals. In this embodiment, the signal processing device 10 is described as being used for verifying whether or not knocking occurs in an internal combustion engine such as a gasoline engine. However, the signal processing device 10 is not limited to this use and can be used for various other uses.

[0020] As shown in FIG. 1, the signal processing system 100 determines whether or not the engine 1 under test is knocking, and includes a sound pressure sensor 4, an in-cylinder pressure sensor 5, a data collection device 6, a monitor 7, headphones 8, a level designation unit 9, and a signal processing device 10.

[0021] As shown in FIG. 1, the engine 1 to be tested is mounted on a vehicle 3. The engine 1 to be tested may be used in a state where it is not mounted on the vehicle 3, for example, in a standalone state. An engine ECU (Electronic Control Unit) 2 that controls the operation of the engine 1 is connected to the engine 1. The engine ECU 2 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random access memory), and other storage devices. The engine ECU 2 controls the operation of the engine 1 while acquiring various information required for the operation control of the engine 1 from outside the engine ECU 2 by processing programs stored in the ROM and storage devices using the CPU.

[0022] The engine ECU 2 outputs angle information representing the current rotation angle of the engine 1 to the data collecting device 6. The angle information includes, for example, a reference angle pulse and a crank angle pulse. The reference angle pulse is a reference pulse output at a reference angle position during one rotation of the crankshaft, and for example, one pulse is output every time the crankshaft rotates once. The crank angle pulse is a pulse output every time the crankshaft rotates at a fixed angle, and for example, when one pulse is output every 1°, 720 pulses are output during two rotations of the crankshaft in one cycle in a four-stroke engine in which four strokes of intake, compression, combustion, and exhaust are one cycle. Note that, if the angle information is input to the data collecting device 6, angle information from an origin sensor that detects the origin of the rotation angle of the crankshaft or an angle sensor that detects the rotation angle of the crankshaft may be input to the data collecting device 6 without going through the engine ECU 2.

[0023] A sound pressure sensor 4 is installed near the engine 1. The sound pressure sensor 4 detects sound generated from the engine 1 and outputs a sound pressure signal based on the detected sound to the data collection device 6. In detail, the sound pressure sensor 4 detects sound pressure, which is an example of a physical quantity based on in-cylinder pressure fluctuations generated in the engine 1, and generates a sound pressure signal indicating the magnitude of the detected sound pressure. Therefore, when the engine 1 is not knocking, the sound pressure signal output from the sound pressure sensor 4 does not include a sound correlated with knocking. On the other hand, when the engine 1 is knocking, the sound pressure signal output from the sound pressure sensor 4 includes a sound correlated with knocking.

[0024] An in-cylinder pressure sensor 5 that detects the in-cylinder pressure of the engine 1 is attached to the engine 1. The in-cylinder pressure sensor 5 outputs an in-cylinder pressure signal including a waveform component corresponding to the combustion gas vibration in the cylinder to a data collection device 6. Here, when the engine 1 is not knocking, the in-cylinder pressure signal output from the in-cylinder pressure sensor 5 does not include a component correlated with the knocking. On the other hand, when the engine 1 is knocking, the in-cylinder pressure signal output from the in-cylinder pressure sensor 5 includes a component correlated with the knocking. The in-cylinder pressure sensor 5 may be integrated with the ignition plug, or may be configured separately from the ignition plug.

[0025] The data collecting device 6 receives the sound pressure signal from the sound pressure sensor 4 and performs A / D conversion. The data collecting device 6 also obtains current angle information from the engine ECU 2 at the timing of receiving the sound pressure signal. The data collecting device 6 then obtains sound pressure signals for one cycle of the engine 1 based on the angle information. Thus, in the case of a four-stroke single cylinder engine, the data collecting device 6 generates sound pressure signals in a number according to the rotation speed of the engine 1 per unit time, for example, 1500 sound pressure signals per minute if the rotation speed is 3000 r / min. The data collecting device 6 also associates angle information with some or all of the sound pressure signals. The data collecting device 6 then outputs the sound pressure signals associated with the angle information to the signal processing device 10. The data collecting device 6 may temporarily hold or temporarily store the sound pressure signals before outputting them to the signal processing device 10. The data collecting device 6 may also associate time information with the sound pressure signals.

[0026] The monitor 7 displays the knocking sound of the engine 1 extracted by the signal processing device 10 and the result of the determination of the presence or absence of knocking by the signal processing device 10. An example of the monitor 7 is a general flat panel display.

[0027] The headphones 8 are a sound emitting unit that emits sound. The headphones 8 are worn on the head of an operator of the signal processing device 10 when inspecting the condition of a test object (in this embodiment, the engine 1) in a sensory test mode described later.

[0028] The level designation unit 9 is an input unit that receives level designation information that designates an increase or decrease level for a signal. The level designation unit 9 is configured with, for example, a touch panel display, a numeric keypad, a dedicated switch, etc. In this embodiment, when the level of an extraction signal (extracted knocking sound 91a (FIG. 11B)) is changed to generate a level-changed extraction signal (level-changed knocking sound 91aa (FIG. 12A)), the level designation unit 9 is operated by the operator of the signal processing device 10 or a person around the operator, including an inspector, in order to designate (input) the increase or decrease level of the extraction signal.

[0029] The signal processing device 10 learns the weights W (FIG. 3B) of a neural network 94 (FIG. 3A) that generates a mask α. Hereinafter, the "neural network 94 that generates a mask α" may be referred to as the "mask generation network 94A (FIG. 3A)." Also, the "weights W of the mask generation network 94A" may be referred to as the "weights W of the neural network." Here, the mask α is a matrix of real or complex numbers for removing noise components from input physical quantities that contain noise components. The mask α changes in accordance with the input physical quantities.

[0030] The signal processing device 10 uses the mask α generated by the trained neural network to extract a knocking sound from sounds in the vicinity of the engine 1, and determines the presence or absence of knocking based on the extracted knocking sound. Then, the signal processing device 10 displays the extracted knocking sound and the determination result of the presence or absence of knocking on the monitor 7.

[0031] Here, the signal processing device 10 is composed of a CPU, a ROM, a RAM, other storage devices, etc. The signal processing device 10 performs arithmetic processing of programs stored in the ROM and storage devices using the CPU. Note that the signal processing device 10 may be a personal computer (PC) or the like having a program that executes the following processes.

[0032] In this embodiment, the signal processing device 10 operates in five operation modes, namely, a learning mode, a threshold calculation mode, a judgment mode, a separation mode, and a sensory test mode. The first learning mode is an operation mode for learning the weight W (FIG. 3B) of the neural network that generates the mask α (FIG. 3A). The second threshold calculation mode is an operation mode for calculating a threshold for judging the presence or absence of knocking after learning the weight W of the neural network that generates the mask α. The third judgment mode is an operation mode for extracting an extraction signal (in this embodiment, an extracted knocking sound 91a) from an input physical quantity (in this embodiment, an engine vicinity sound 90) using a mask α generated by inputting an input physical quantity to the trained neural network, and judging the presence or absence of knocking. The fourth separation mode is an operation mode for separating the input physical quantity into an extraction signal and a noise component (in this embodiment, noise 91b). The fifth sensory test mode is an operation mode for an inspector to listen to a processed sound described later and determine data to be used in the threshold calculation mode for a target sound to be inspected (in this embodiment, a knocking sound) by the inspector's sense of hearing. In the fifth sensory test mode, for example, a threshold is calculated from sounds that fall outside the acceptable range.

[0033] These five operation modes can be arbitrarily switched. For example, the operation modes of the signal processing device 10 can be switched by a management device (not shown) via a CAN (Controller Area Network). The operation modes of the signal processing device 10 may also be switched by using operation means such as a mouse and a keyboard (not shown).

[0034] In the learning mode, the engine 1 is operated under operating conditions where knocking occurs and operating conditions where knocking does not occur, and the data collection device 6 collects the in-cylinder pressure signal as teacher data (teacher signal). At this time, the data collection device 6 receives the in-cylinder pressure signal from the in-cylinder pressure sensor 5, A / D converts it, and associates it with the sound pressure signal.

[0035] In the case of the threshold calculation mode, the engine 1 is operated under operating conditions in which knocking does not occur, and the data collecting device 6 collects sound pressure signals.

[0036] In the determination mode, the mask α generated by the trained neural network is used to extract the extracted knocking sound 91a (extracted signal) from the near-engine sound 90 (input physical quantity), and the signal processing device 10 determines the presence or absence of knocking based on a threshold value.

[0037] In addition, in the case of the threshold calculation mode or the judgment mode, it is not necessary for the data collection device 6 to collect the in-cylinder pressure signal. That is, in the case of the learning mode, both the sound pressure sensor 4 and the in-cylinder pressure sensor 5 operate, but in the case of the threshold calculation mode or the judgment mode, only the sound pressure sensor 4 operates.

[0038] In the separation mode, the engine 1 is operated, and the signal processing device 10 inputs the engine vicinity sound 90 (input physical quantity) to the learned neural network to generate a mask α, and separates the sound into noise 91b (noise component) and extracted knocking sound 91a (extracted signal). At this time, the signal processing device 10 stores the noise 91b in the noise component storage unit 26c and the extracted knocking sound 91a in the extracted signal storage unit 26d. In the present invention, the separation performance is improved by learning the weight W of the neural network that generates the mask α, taking into account the amplitude and phase related to the input physical quantity. The mask α is implemented as a real number or a complex number. This allows the signal processing device 10 to satisfactorily separate the input physical quantity into the noise component and the extracted signal.

[0039] In the sensory test mode, the signal processing device 10 receives level designation information from the level designation unit 9, and generates a processed sound using the noise 91b (noise component) and the extracted knocking sound 91a (extracted signal) based on the level designation information.

[0040] Even if the amplitude spectrum is the same, the signal waveform changes significantly depending on the phase spectrum, which has a significant impact on the auditory impression. Therefore, in order to generate a processed sound with good quality, it is important to acquire an extracted signal (in this embodiment, the extracted knocking sound 91a) to be used for the processed sound, taking into account the phase. In other words, the signal processing device 10 improves separation performance by learning the weight W of the neural network that generates the mask α, taking into account the amplitude and phase related to the input physical quantity, and can generate a processed sound with good quality (a processed sound in which the knocking sound is natural to the auditory sense).

[0041] <Configuration of signal processing device> The configuration of the signal processing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the signal processing device 10. As shown in Fig. 2, the signal processing device 10 includes a signal extraction unit 11, a spectrogram calculation unit 12, a signal storage unit 13, a switch 14, a learning processing unit 20 (learning device), a judgment processing unit 30, a separation unit 40, and a signal synthesis unit 50. Here, a sound pressure signal and angle information associated with this sound pressure signal are input from a data collection device 6 to the signal processing device 10. Furthermore, in the case of a learning mode, an in-cylinder pressure signal is input from the data collection device 6 to the signal processing device 10.

[0042] The signal cutting unit 11 cuts out a sound pressure signal in a predetermined cutting angle range from the input sound pressure signal based on the angle information input from the data collecting device 6. For example, the cutting angle range is an angle range of about -10 to 90° of ATDC (After Top Dead Center). In this embodiment, since the cutting is performed based on TDC (Top Dead Center), the cutting angle range remains fixed even if the ignition timing is changed. However, the cutting angle range may be changed according to the change in the ignition timing. When the signal cutting unit 11 cuts out the sound pressure signal, it outputs the cut sound pressure signal to the spectrogram calculating unit 12.

[0043] The spectrogram calculation unit 12 calculates a spectrogram of the sound pressure signal by performing a Short Time Fourier Transform (STFT) on the sound pressure signal extracted by the signal extraction unit 11. The Short Time Fourier Transform is performed by, for example, a Fast Fourier Transform (FFT) that quickly calculates a Discrete Fourier Transform. Thereafter, the spectrogram calculation unit 12 writes the spectrogram of the sound pressure signal into the signal storage unit 13.

[0044] The signal storage unit 13 is a storage device such as a memory, HDD (Hard Disk Drive), or SSD (Solid State Drive) that stores a spectrogram of the sound pressure signal converted by the spectrogram calculation unit 12. In the learning mode, the signal storage unit 13 stores the in-cylinder pressure signal (observed knocking in-cylinder pressure 93 (FIG. 3A)) input from the data collection device 6 as teacher data (teacher signal). Here, the "knocking in-cylinder pressure" refers to the knocking component superimposed on the in-cylinder pressure signal. The knocking in-cylinder pressure is obtained by subjecting the in-cylinder pressure to a high-pass filter process that passes a frequency band equal to or higher than the frequency component of knocking.

[0045] The switch 14 can be arbitrarily switched to a learning mode connection part M1, a threshold calculation mode connection part M2, a determination mode connection part M3, a separation mode connection part M4, or a sensory test mode connection part M5, which correspond to the five operation modes described above. The signal processing device 10 can output a signal stored in the signal storage part 13 to an arbitrary output destination by switching the switch 14 according to the operation mode of the signal processing device 10.

[0046] For example, in the learning mode, the signal processing device 10 connects the switch 14 to a connection section M1 for learning mode and outputs the spectrogram of the sound pressure signal and the in-cylinder pressure signal stored in the signal storage section 13 to a learning section 21 described later. In the threshold calculation mode, the signal processing device 10 connects the switch 14 to a connection section M2 for threshold calculation mode and outputs the spectrogram of the sound pressure signal stored in the signal storage section 13 to a first estimation section 23 described later. In the determination mode, the signal processing device 10 connects the switch 14 to a connection section M3 for determination mode and outputs the spectrogram of the sound pressure signal stored in the signal storage section 13 to a second estimation section 31 (extracted signal estimation section) described later. In the separation mode, the signal processing device 10 connects the switch 14 to a connection section M4 for separation mode and outputs the spectrogram of the sound pressure signal stored in the signal storage section 13 to a separation section 40 described later. In addition, in the case of the sensory test mode, the signal processing device 10 connects the switch 14 to the sensory test mode connection section M5 and outputs an execution instruction signal for the sensory test mode, which is generated in advance by a signal generation section (not shown) and stored in the signal storage section 13, to a signal adjustment section 51 (described later). Here, the execution instruction signal for the sensory test mode is described as being generated in advance by a signal generation section (not shown) and stored in the signal storage section 13. However, the execution instruction signal for the sensory test mode may be generated by a signal generation section (not shown) when the switch 14 is connected to the sensory test mode connection section M5, and output from the signal generation section (not shown) to the signal adjustment section 51 (described later) without passing through the signal storage section 13.

[0047] <Learning processing unit> In the learning mode, the learning processing unit 20 performs a learning process (FIG. 14A) for learning the weight W (FIG. 3B) of the neural network that generates the mask α, and in the threshold calculation mode, performs a threshold calculation process (FIG. 15A) for calculating a threshold used for threshold determination of the extracted knocking sound of the engine 1. As shown in FIG. 2, the learning processing unit 20 includes a learning unit 21, a learned parameter storage unit 22, a first estimation unit 23, a threshold calculation unit 24, a threshold storage unit 25, a teacher signal storage unit 26a, a noise component storage unit 26c, and an extracted signal storage unit 26d.

[0048] In the learning mode, the learning unit 21 learns the weights W of a neural network 94 ( FIG. 3A ) that generates a mask α that removes noise 91b from the near-engine sound 90 and extracts the extracted knocking sound 91a. In this embodiment, the learning unit 21 learns the weights W of the neural network 94 that generates the mask α, using a spectrogram of the sound pressure signal and an in-cylinder pressure signal input from the signal storage unit 13. Thereafter, the learning unit 21 writes the learned weights W of the neural network 94 that generates the mask α to the learned parameter storage unit 22.

[0049] The learned parameter storage unit 22 is a storage device such as a memory, HDD, or SSD that stores learned parameters (weights W of the neural network that generate the mask α).

[0050] In the threshold calculation mode, the first estimation unit 23 extracts an extracted knocking sound 91a of the engine 1 from the spectrogram of the sound pressure signal by using the mask α generated by the neural network 94. The extracted knocking sound 91a extracted by the first estimation unit 23 is used when calculating a threshold value described later. Specifically, the first estimation unit 23 inputs the spectrogram of the sound pressure signal input from the signal storage unit 13 to the neural network 94 reflecting the weight W of the neural network that generates the mask α of the learned parameter storage unit 22. Then, the mask α generated by the learned neural network extracts the extracted knocking sound 91a from the spectrogram of the sound pressure signal. After that, the first estimation unit 23 outputs the extracted knocking sound 91a to the threshold calculation unit 24.

[0051] The threshold calculation unit 24 calculates a threshold based on the extracted knocking sound 91a of the engine 1 extracted by the first estimation unit 23. Specifically, the threshold calculation unit 24 sums up the absolute values ​​of the spectrogram of the extracted knocking sound 91a for each predetermined time (for example, one cycle of the engine 1). For example, in the engine 1 having a plurality of cylinders, when the extracted knocking sounds 91a are summed up, one score is obtained for each cylinder. Next, the threshold calculation unit 24 calculates the median of the extracted knocking sounds 91a summed up for each predetermined time. For example, the threshold calculation unit 24 calculates the median of the sum of the extracted knocking sounds 91a for all cycles. At this time, the threshold calculation unit 24 adds a preset margin of any value to the median to obtain the threshold. The threshold calculation unit 24 may sum up the extracted knocking sounds 91a for each cylinder to obtain the median, and calculate the threshold for each cylinder. The threshold calculation unit 24 may also sum up the extracted knocking sounds 91a for each cylinder, find the median value for all cylinders, and calculate a common threshold for all cylinders. Alternatively, the sum of the extracted knocking sounds 91a for the processed sounds 91c that are determined to be unacceptable by an inspector in a sensory test described later may be calculated, and any value equal to or less than the sum may be set as the threshold. Thereafter, the threshold calculation unit 24 writes the calculated threshold into the threshold storage unit 25.

[0052] The threshold storage unit 25 is a storage device such as a memory, HDD, or SSD that stores the threshold calculated by the threshold calculation unit 24.

[0053] The teacher signal storage unit 26a is a storage device such as a memory, HDD, or SSD that stores a teacher signal (in this embodiment, the observed knocking in-cylinder pressure 93 (FIG. 3A)).

[0054] The noise component storage unit 26c is a storage device such as a memory, HDD, or SSD that stores noise components (in this embodiment, noise 91b (FIG. 6A)) included in the input physical quantity.

[0055] The extracted signal storage unit 26d is a storage device such as a memory, HDD, or SSD that stores an extracted signal (in this embodiment, an extracted knocking sound 91a (FIG. 6A)) obtained by removing the noise component from the input physical quantity including the noise component.

[0056] <Determination processing unit> In the determination mode, the determination processing unit 30 extracts the knocking sound of the engine 1 from the spectrogram of the sound pressure signal and performs a determination process (FIG. 16) to determine whether knocking occurs. As shown in FIG. 2, the determination processing unit 30 includes a second estimation unit 31 (extracted signal estimation unit) and a threshold determination unit 32.

[0057] In the determination mode, the second estimation unit 31 (extracted signal estimation unit) extracts an extracted knocking sound 91a (extracted signal) from the spectrogram of the sound pressure signal using a mask α generated by the neural network 94. The extracted knocking sound 91a extracted by the second estimation unit 31 is used for threshold determination, which will be described later. Note that the processing content of the second estimation unit 31 is similar to that of the first estimation unit 23, and therefore a description thereof will be omitted. Thereafter, the second estimation unit 31 outputs the extracted knocking sound 91 a to the threshold determination unit 32 .

[0058] In the determination mode, the threshold determination unit 32 determines the presence or absence of knocking by threshold determination between the threshold stored in the threshold storage unit 25 and the extracted knocking sound 91a extracted by the second estimation unit 31. Specifically, like the threshold calculation unit 24, the threshold determination unit 32 sums up the absolute values ​​of the spectrogram of the extracted knocking sound 91a at predetermined time intervals. Then, the threshold determination unit 32 compares the summed extracted knocking sound 91a with the threshold, and determines that knocking is present when the summed extracted knocking sound 91a exceeds the threshold, and determines that knocking is not present when the summed extracted knocking sound 91a is equal to or less than the threshold. Thereafter, the threshold determination unit 32 outputs the determination result of the presence or absence of knocking and the extracted knocking sound 91a input from the second estimation unit 31 to the monitor 7 (FIG. 1).

[0059] <Separation part> The separation unit 40 separates the input physical quantity into a noise component and an extracted signal by using the mask α. In this embodiment, as shown in Fig. 6A, the separation unit 40 separates a near-engine sound 90 as an input physical quantity into noise 91b, which is a noise component, and an extracted knocking sound 91a, which is an extracted signal.

[0060] <Signal synthesis section> The signal synthesis unit 50 synthesizes various signals to generate a processed sound. In this embodiment, as shown in Figs. 11A to 12C, the signal synthesis unit 50 changes the level of the extracted knocking sound 91 (extracted signal) and synthesizes it with noise 91b (noise components) separated from the engine vicinity sound 90 (input physical quantity) to generate a processed sound. The signal synthesis unit 50 has a signal adjustment unit 51 and a signal output unit 52. The signal adjustment unit 51 changes the level of the extracted knocking sound 91 according to the rising or falling level of the signal designated (input) by the level designation unit 9, and synthesizes the extracted knocking sound 91 (level-changed extracted signal) whose level has been changed with the noise components to generate a processed sound. The signal output unit 52 outputs the processed sound (signal) to the sound emission unit (headphones 8) to cause the sound emission unit to emit the processed sound.

[0061] <Learning mode operation> The operation of the signal processing device 10 in the learning mode will be described with reference to Figs. 3A, 3B, 8A, 8B, and 9. Fig. 3A is an explanatory diagram of the learning mode. Fig. 3B is an explanatory diagram of the operation of the signal processing device 10 in the learning mode. Fig. 8A is an explanatory diagram showing the relationship between a noise component and a teacher signal during learning. Fig. 8B is an explanatory diagram showing the relationship between an extracted signal and a teacher signal during learning. Fig. 9 is an explanatory diagram of learning the weight W of a neural network that generates a mask α in an embodiment.

[0062] As shown in Fig. 3A, in the learning mode, the learning unit 21 of the signal processing device 10 separates the near-engine sound 90 (input physical quantity) into noise 91b (noise component) and an extracted knocking sound 91a (extracted signal). At this time, the learning unit 21 multiplies the near-engine sound 90 by a mask α to obtain the extracted knocking sound 91a. The learning unit 21 also obtains the noise 91b by subtracting the extracted knocking sound 91a (extracted signal) from the near-engine sound 90. In this embodiment, the mask α takes into consideration the proportion and phase component of the knocking sound contained in the near-engine sound 90. The phase component will be described later with reference to Figs. 10A and 10B.

[0063] When learning the weight W of the neural network that generates the mask α, the learning unit 21 of the signal processing device 10 learns the mask α by taking into account the amplitude and phase components related to the near-engine sound 90 (input physical quantity). The phase components will be described later with reference to Figs. 10A and 10B.

[0064] In this embodiment, as shown in FIG. 3A, the learning unit 21 learns the weights of the neural network so that the correlation between the noise 91b (noise component) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced and the correlation between the extracted knocking sound 91a (extracted signal) and the teacher signal (observed knocking in-cylinder pressure 93) is increased. Specifically, the learning unit 21 learns the weights W of the neural network that generates the mask α so that the coherence between the first signal Sfi acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise 91b (noise component) and the observed knocking in-cylinder pressure 93 (teacher signal) is reduced and the coherence between the second signal Sse acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted knocking sound 91a (extracted signal) and the observed knocking in-cylinder pressure 93 (teacher signal) is increased. This allows the signal processing device 10 to remove the sound caused by the in-cylinder pressure from the noise 91b.

[0065] Fig. 3B shows the operation of the signal processing device 10 in the learning mode. The bold frame in Fig. 3B indicates the components that operate in the learning mode. The bold arrow in Fig. 3B indicates the signals that are output in the learning mode.

[0066] As shown in FIG. 3B, in the learning mode, the signal processing device 10 connects the switch 14 to the learning mode connection unit M1 and outputs the spectrogram of the near-engine sound 90 and the observed knocking in-cylinder pressure 93 stored in the signal storage unit 13 to the learning unit 21.

[0067] In response to this, the learning unit 21 learns the weights W of a neural network 94 (FIG. 3A) that generates a mask α that removes noise 91b (noise component) from the near-engine sound 90 (input physical quantity). At this time, the learning unit 21 generates a mask α that corresponds to the near-engine sound 90 every time a new near-engine sound 90 is input.

[0068] Then, the learning unit 21 stores the learned parameters (weight W of the neural network that generates the mask α) in the learned parameter storage unit 22. The learning unit 21 also stores the observed knocking in-cylinder pressure 93 in the teacher signal storage unit 26a.

[0069] <Operation in threshold calculation mode> The operation of the signal processing device 10 in the threshold calculation mode will be described with reference to Figures 4A and 4B. Figure 4A is an explanatory diagram of the threshold calculation mode. Figure 4B is an explanatory diagram of the operation of the signal processing device 10 in the threshold calculation mode.

[0070] As shown in Fig. 4B, in the threshold calculation mode, unlike the learning mode in Fig. 3B, the learning unit 21 of the signal processing device 10 is in a stopped state. Instead, as shown in Fig. 4B, the first estimation unit 23 and the threshold calculation unit 24 of the learning processing unit 20 operate to calculate the threshold T used for threshold determination of the extracted knocking sound 91a of the engine 1. The bold frame in Fig. 4B indicates the components that operate in the threshold calculation mode. Also, the bold arrow in Fig. 4B indicates the signals that are output in the threshold calculation mode.

[0071] As shown in FIG. 4B, in the threshold calculation mode, the signal processing device 10 connects the switch 14 to the threshold calculation mode connection unit M2 and outputs the spectrogram of the near-engine sound 90 stored in the signal storage unit 13 to the first estimation unit 23.

[0072] In response to this, the first estimation unit 23 acquires the weight W of the neural network that generates the mask α from the learned parameter storage unit 22. Then, the first estimation unit 23 extracts an extracted knocking sound 91a of the engine 1 from the near-engine sound 90 using the mask α generated by the neural network 94 (FIG. 4A). FIG. 4A shows an overview of the operation of the first estimation unit 23 at this time. Thereafter, the first estimation unit 23 outputs the extracted knocking sound 91a to the threshold calculation unit 24.

[0073] In response to this, the threshold calculation unit 24 sums up the absolute values ​​of the spectrograms of the extracted knocking sound 91a in each cycle of the engine 1, and adds a preset margin to the sum to calculate the threshold T. Then, the threshold calculation unit 24 stores the threshold T in the threshold storage unit 25.

[0074] <Operation in judgment mode> The operation of the signal processing device 10 in the determination mode will be described with reference to Figures 5A and 5B. Figure 5A is an explanatory diagram of the determination mode. Figure 5B is an explanatory diagram of the operation of the signal processing device 10 in the determination mode.

[0075] As shown in Fig. 5B, in the judgment mode, unlike the learning mode in Fig. 3B, the learning unit 21 of the signal processing device 10 is in a stopped state. Instead, as shown in Fig. 5B, the second estimation unit 31 (extracted signal estimation unit) and the threshold judgment unit 32 of the judgment processing unit 30 operate to judge the presence or absence of knocking. The bold frame in Fig. 5B indicates the components that operate in the judgment mode. Also, the bold arrow in Fig. 5B indicates the signals that are output in the judgment mode.

[0076] As shown in FIG. 5B, in the determination mode, the signal processing device 10 connects the switch 14 to the determination mode connection unit M3 and outputs the spectrogram of the near-engine sound 90 stored in the signal storage unit 13 to the second estimation unit 31.

[0077] In response to this, the second estimation unit 31 acquires the weight W of the neural network that generates the mask α from the learned parameter storage unit 22. Then, the second estimation unit 31 extracts an extracted knocking sound 91a from the near-engine sound 90 using the mask α generated by the neural network 94 (FIG. 5A). FIG. 5A shows an overview of the operation of the second estimation unit 31 at this time. Thereafter, the second estimation unit 31 outputs the extracted knocking sound 91a to the threshold determination unit 32.

[0078] In response to this, the threshold determination unit 32 obtains the threshold T from the threshold storage unit 25, and determines the presence or absence of knocking by comparing the sum of the absolute values ​​of the extracted knocking sounds 91a with the threshold T. Then, the threshold determination unit 32 outputs, for example, the determination result of the presence or absence of knocking and a waveform diagram showing the relationship between the extracted knocking sound 91a and the threshold T to the monitor 7 for display.

[0079] <Operation in separate mode> The operation of the signal processing device 10 in the separation mode will be described with reference to Figures 6A and 6B. Figure 6A is an explanatory diagram of the separation mode. Figure 6B is an explanatory diagram of the operation of the signal processing device 10 in the separation mode.

[0080] As shown in Fig. 6B, in the separation mode, unlike the learning mode in Fig. 3B, the learning unit 21 of the signal processing device 10 is in a stopped state. Instead, as shown in Fig. 6B, the separation unit 40 operates to separate the near-engine sound 90 into an extracted knocking sound 91a and noise 91b. The bold frame in Fig. 6B indicates the components that operate in the separation mode. Also, the bold arrow in Fig. 6B indicates the signals that are output in the separation mode.

[0081] As shown in FIG. 6B, in the separation mode, the signal processing device 10 connects the switch 14 to the separation mode connection section M4 and outputs the spectrogram of the near-engine sound 90 stored in the signal storage section 13 to the separation section 40.

[0082] In response to this, the separation unit 40 acquires the weight W of the neural network that generates the mask α from the learned parameter storage unit 22. Then, the separation unit 40 separates the near-engine sound 90 into an extracted knocking sound 91a and noise 91b using the mask α generated by the neural network 94 (FIG. 6A). FIG. 6A shows an overview of the operation of the separation unit 40 at this time. Thereafter, the separation unit 40 stores the extracted knocking sound 91a in the extracted signal storage unit 26d, and stores the noise 91b in the noise component storage unit 26c.

[0083] <Operation in sensory test mode> The operation of the signal processing device 10 in the sensory test mode will be described with reference to Fig. 7A and Fig. 7B. Fig. 7A is an explanatory diagram of the sensory test mode. Fig. 7B is an explanatory diagram of the operation of the signal processing device 10 in the sensory test mode. The sensory test mode is a mode for calculating a threshold value T based on the auditory sense of the examiner.

[0084] As shown in Fig. 7B, in the sensory test mode, unlike the learning mode in Fig. 3B, the learning unit 21 of the signal processing device 10 is in a stopped state. Instead, as shown in Fig. 7B, the signal adjustment unit 51 and the signal output unit 52 of the signal synthesis unit 50, and the first estimation unit 23 and the threshold calculation unit 24 of the learning processing unit 20 operate to calculate a threshold T based on the auditory sense of the examiner. The bold frame in Fig. 7B indicates the components that operate in the sensory test mode. Also, the bold arrow in Fig. 7B indicates the signal output in the sensory test mode.

[0085] 7B, in the sensory test mode, the signal processing device 10 connects the switch 14 to the sensory test mode connection section M5, thereby outputting an execution instruction signal for the sensory test mode, which is generated in advance by a signal generation section (not shown) and stored in the signal storage section 13, to the signal adjustment section 51. However, the execution instruction signal for the sensory test mode may be generated by a signal generation section (not shown) when the switch 14 is connected to the sensory test mode connection section M5, and output from the signal generation section (not shown) to the signal adjustment section 51 without passing through the signal storage section 13.

[0086] In response to this, the signal adjustment unit 51 receives level designation information from the level designation unit 9, and acquires the extracted knocking sound 91a stored in the extracted signal storage unit 26d and the noise 91b stored in the noise component storage unit 26c. Then, the signal adjustment unit 51 generates a processed sound 91c using the extracted knocking sound 91a and the noise 91b based on the level designation information. At this time, the signal adjustment unit 51 raises or lowers the level (volume) of the extracted knocking sound 91a by an amount designated by the level designation information, and then combines it with the noise 91b to generate the processed sound 91c and outputs it to the signal output unit 52. The signal output unit 52 outputs the processed sound 91c to the headphones 8 (sound output unit) to emit the sound.

[0087] Further, the signal adjustment unit 51 outputs the processed sound 91c to the first estimation unit 23. The first estimation unit 23 acquires the weight W of the neural network that generates the mask α from the learned parameter storage unit 22. Then, the first estimation unit 23 extracts the extracted knocking sound 91a from the processed sound 91c using the mask α generated by the neural network 94 (FIG. 7A). FIG. 7A shows an overview of the operation of the first estimation unit 23 at this time. After that, the first estimation unit 23 outputs the extracted knocking sound 91a to the threshold calculation unit 24. The threshold calculation unit 24 sums up the absolute values ​​of the spectrograms of the extracted knocking sound 91a in each cycle of the engine 1, and sets an arbitrary value equal to or less than the sum value as the threshold T. Then, the threshold calculation unit 24 stores the threshold T in the threshold storage unit 25.

[0088] Incidentally, the conventional techniques described in Patent Documents 2 and 3 do not include a function for measuring the degree of separation of sounds in the objective function during learning of the neural network, so there is a possibility that the knocking sound (target sound) may be mixed into the noise (sounds other than the knocking sound (background sound)) removed from the engine vicinity sound (input physical quantity). In other words, the conventional techniques described in Patent Documents 2 and 3 only minimize the square error between the estimated in-cylinder pressure and the measured in-cylinder pressure (teacher data) during learning, so they do not monitor whether the "engine sound from which noise has been removed" is one from which only noise has been removed satisfactorily. For example, the conventional technique described in Patent Document 3 applies a mask α that does not take into account the phase component related to the engine vicinity sound (input physical quantity) to the engine vicinity sound to extract the "engine sound from which noise has been removed," i.e., the knocking sound (corresponding to the "extracted signal" in this embodiment). At that time, the conventional technique described in Patent Document 3 does not use a function for measuring the degree of separation of sounds, so there is a possibility that the knocking sound (target sound) that should not be removed may be removed from the engine vicinity sound together with the noise. Therefore, the conventional technique described in Patent Document 3 may degrade the performance of evaluating the presence or absence of knocking.

[0089] In contrast, the signal processing device 10 according to the present embodiment is configured to evaluate whether or not a knocking sound (target sound) is mixed in the noise removed from the sound near the engine (input physical quantity) during learning. To achieve this, the signal processing device 10 according to the present embodiment is configured to learn so that the correlation between the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced. Specifically, as shown in FIG. 3A, learning is performed so that the coherence between the first signal Sfi acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced. In addition, the signal processing device 10 according to the present embodiment is configured to learn so that the correlation between the extracted knocking sound 91a (extracted signal) and the teacher signal (observed knocking in-cylinder pressure 93) is increased. 3A, learning is performed so that the coherence between the second signal Sse acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted knocking sound 91a (extracted signal) and the teacher signal (observed knocking in-cylinder pressure 93) becomes large. The signal processing device 10 according to this embodiment can improve the evaluation performance of the presence or absence of knocking compared to the conventional techniques described in Patent Documents 2 and 3.

[0090] Fig. 8A shows a spectrogram F11 of the noise 91b (noise component), a signal waveform F12 of the first signal Sfi (Fig. 3A) obtained by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram F11, and a signal waveform F93 of the observed knocking in-cylinder pressure 93 (teacher signal). Fig. 8A also shows coherence F13 that is minimized in the process of learning the signal waveform F12 of the first signal Sfi (Fig. 3A) and the signal waveform F93 of the observed knocking in-cylinder pressure 93 (teacher signal).

[0091] Fig. 8B shows a spectrogram F21 of the extracted knocking sound 91a (extracted signal), a signal waveform F22 of the second signal Sse (Fig. 3A) obtained by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram F21, and a signal waveform F93 of the observed knocking in-cylinder pressure 93 (teacher signal). Fig. 8B also shows coherence F23 that is maximized in the learning process between the signal waveform F22 of the second signal Sse (Fig. 3A) and the signal waveform F93 of the observed knocking in-cylinder pressure 93 (teacher signal).

[0092] Coherence is defined by the following equation (1): Coherence function γ 2 indicates the degree of correlation between the input and output of the system. The coherence function γ 2 is the square of the absolute value of the cross spectrum divided by the power spectrum of each of the measured input and the system output.

number

[0093] Here, Wxy is the average value of the cross spectrum, Wxx is the average value of the power spectrum of x, and Wyy is the average value of the power spectrum of y. Coherence function γ 2 takes values ​​between 0 and 1. γ 2 When (f) is 1, it indicates that the system output at that frequency f is entirely due to the measured input. Also, γ 2 If (f) is 0, it means that the output of the system is completely independent of the measured input at that frequency f. Also, 0<γ 2 If (f)<1, it is likely that there is a signal unrelated to the measurement, noise occurring within the system, nonlinearity of the system, etc.

[0094] Alternatively, the coherent output power may be used as a method for evaluating whether or not a knocking sound (target sound) is mixed into the noise removed from the sound near the engine (input physical quantity). The learning unit 21 learns the weight W of the neural network that generates the mask α so that the coherence or coherent output power between a signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on a noise component (noise 91b) and a teacher signal (observed knocking in-cylinder pressure 93) is reduced, and the coherence or coherent output power between a signal waveform obtained by performing an inverse short-time Fourier transform (ISTFT) on an extracted signal (extracted knocking sound 91a) and a teacher signal (observed knocking in-cylinder pressure 93) is increased. This allows the signal processing device 10 to further improve the evaluation performance for the presence or absence of knocking.

[0095] In addition, for coherence and coherent output power, when the evaluation objects are the following two sets of "I" and "II" and the evaluation methods are the following two sets of "A" and "B", the signal processing device 10 enables evaluation of the following four evaluation patterns. (Subject to evaluation) I: Noise components and teacher signals II: Extracted signals and teacher signals (Evaluation method) A: Coherence B: Coherent output power (Evaluation pattern) 1: Evaluate I with an A, and II with an A. 2: Evaluate I with B, Evaluate II with B. 3: Evaluate I with A and II with B. 4: Evaluate I with a B and II with an A.

[0096] The coherent output power is given by the coherence function γ 2 and the average value Wyy of the power spectrum of y. The coherent output power is the power that is contained in the output of the system due to the input.

[0097] In this embodiment, the learning unit 21 learns the weights W of a neural network that generates the mask α using a convolutional neural network (CNN). Note that the learning unit 21 is not limited to a CNN, and may have other configurations such as a recurrent neural network (RNN) or a Transformer, or may use other machine learning methods.

[0098] As shown in FIG. 9, U-Net95 is an example of a neural network 94 that learns the weights W of the neural network that generates the mask α. This U-Net95 is a type of Encoder-Decoder model, and is a deep learning method used for image recognition and sound separation. In sound separation, U-Net95 performs convolution (stride is 2 or more) in the downward path 96 (Encoder), and extracts sound features as the layer 97 becomes deeper. Meanwhile, in the upward path 98 (Decoder), the mask α is generated by performing deconvolution and UP sampling (dilation) from the extracted sound features. Up to this point, the configuration of a general Encoder-Decoder model. Furthermore, in U-Net95, the output 99 from the convolution layer of each Encoder is merged into the convolution layer of the Decoder. As a result, U-Net95 can generate a mask α with higher accuracy than a general Encoder-Decoder model. The signal processing device 10 can obtain an extracted signal (extracted knocking sound 91a) by generating a mask α and multiplying it by the near-engine sound 90. The mask α is a spatial mask represented by frequency and time.

[0099] In this way, the learning unit 21 learns the weights W of the neural network that generates the mask α using U-Net 95, so that the trained neural network generates an appropriate mask α according to the input sound in the vicinity of the engine 1. Therefore, the signal processing device 10 can extract the knocking sound of the engine 1 well.

[0100] <Influence of phase components> The influence of phase components will be described with reference to Figures 10A and 10B. Figure 10A is an explanatory diagram of a calculation example in the case where phase components of pressure, vibration, sound, etc. are not taken into consideration. Figure 10B is an explanatory diagram of a calculation example in the case where phase components of pressure, vibration, sound, etc. are taken into consideration. In Figures 10A and 10B, the arrows in the circles represent phase components contained in pressure, vibration, sound, etc.

[0101] In both the examples shown in FIGS. 10A and 10B, the following conditions are assumed: The system is a linear time-invariant system (a system in which the transfer characteristics between the input and output are linear and do not change with time).

[0102] 10A, when the engine 1 is driven, an observed knocking in-cylinder pressure 93 is observed by the in-cylinder pressure sensor 5 (FIG. 1). The observed knocking in-cylinder pressure 93 propagates through the engine 1, causing a knocking sound 82 to be emitted. Furthermore, the knocking sound 82 is combined with mechanical noise 83 (noise component), which is a background sound, and a near-engine sound 90 (input physical quantity) is observed by the sound pressure sensor 4.

[0103] FIG. 10A shows an example of rough calculation of the observed knocking in-cylinder pressure 93, knocking vibration 81, knocking sound 82, mechanical noise 83, and near-engine sound 90 as follows, assuming that the phase φ does not change. Observed knocking cylinder pressure 93 = ASin(Ωt+φ) Knocking vibration 81 = ABSin(Ωt+φ) Knocking sound 82 = ABCSin(Ωt+φ) Mechanical noise 83 = NSin(Ωt+φN) Near-engine noise 90 = (ABC+N)Sin(Ωt+φ)

[0104] 10B is an example in which the phase is taken into consideration. As shown below, an example of a rough calculation of the observed knocking in-cylinder pressure 93, the knocking vibration 81, the knocking sound 82, the mechanical noise 83, and the sound near the engine 90 is shown, and it is taken into consideration that the phase φ changes every time the dimension (unit) changes. Observed knocking cylinder pressure 93 = ASin(Ωt+φA) Knocking vibration 81 = ABSin(Ωt+φA+φB) Knocking sound 82 = ABCSin(Ωt+φA+φB+φC) Mechanical noise 83 = NSin(Ωt+φN) Near-engine noise 90 = ABCSin(Ωt+φA+φB+φC)+NSin(Ωt+φN)

[0105] In the example shown in FIG. 10B, the change in the phase component when the pressure becomes vibration and is emitted as sound is taken into consideration, so that the signal can be analyzed better than in the example shown in FIG. 10A. Therefore, as shown in FIG. 3A, the signal processing device 10 is configured to learn the weight W of the neural network that generates the mask α while taking the phase into consideration. Such a signal processing device 10 can effectively separate the engine vicinity sound 90 (input physical quantity) into the extracted knocking sound 91a (extracted signal) and noise 91b (noise component). In other words, the signal processing device 10 can separate the engine vicinity sound 90 into the extracted knocking sound 91a and noise 91b so that the knocking sound (target sound) is not mixed into the noise. Such a signal processing device 10 can improve the evaluation performance of the presence or absence of knocking compared to the conventional techniques described in Patent Documents 2 and 3. In addition, such a signal processing device 10 can perform a good sensory test.

[0106] (Overview of sensory testing) An overview of the sensory test will be described with reference to Figures 11A to 12C. Figures 11A to 12C are explanatory diagrams of signal processing in the sensory test.

[0107] 2, pressure and sound signals are collected by a data collecting device 6 and output to a signal processing device 10. In the signal processing device 10, a signal extracting unit 11 extracts each signal, a spectrogram calculating unit 12 calculates a spectrogram, and a signal storage unit 13 stores the spectrogram.

[0108] The signal processing device 10 according to this embodiment can execute the separation mode and the sensory test mode. In the separation mode, the signal processing device 10 separates the engine vicinity sound 90 (input physical quantity) into an extracted knocking sound 91a (extracted signal) and noise 91b (noise component). In the sensory test mode, the signal processing device 10 changes the level of the extracted knocking sound 91a to generate a level-changed extracted signal (level-changed knocking sound 91aa (FIG. 12A)), and combines it with the noise 91b to generate a processed sound 91c (FIG. 12C).

[0109] In the sensory test mode, the examiner wears headphones 8 (FIG. 1) on his / her head. Then, the examiner himself / herself or an operator who separately specifies the level operates a level specification unit 9 to input level specification information to the signal processing device 10. The signal processing device 10 changes the level of the extracted knocking sound 91a based on the input level specification information.

[0110] FIG. 11A shows the engine vicinity sound 90 (input physical quantity), and FIG. 11B and FIG. 11C show the extracted knocking sound 91a (extracted signal) and noise 91b (noise component) separated from the engine vicinity sound 90 in the separation mode. FIG. 12A shows the level-changed knocking sound 91aa (level-changed extracted signal) generated by changing the level of the extracted knocking sound 91a (extracted signal) to be increased by 3 dB in the sensory test mode, FIG. 12B shows the noise 91b to be synthesized with the level-changed knocking sound 91aa, and FIG. 12C shows the processed sound 91c obtained by synthesizing the level-changed knocking sound 91aa and the noise 91b. The signal processing device 10 can create a state in which the target sound (knocking sound) to be inspected is easily distinguished by generating the processed sound 91c. Such a signal processing device 10 can allow the inspector to grasp the presence or absence of the target sound (knocking sound) with high accuracy, thereby improving the inspection performance. Furthermore, the signal processing device 10 calculates the sum of the absolute values ​​of the spectrograms of the processed sounds 91c that are determined to be unacceptable by the inspector, and writes an arbitrary value equal to or smaller than the sum in the threshold storage unit 25. This makes the threshold value T a value that matches the senses of the inspector.

[0111] <Operation of the signal processing device> The operation of the signal processing device 10 will be described with reference to Figs. 13 to 18. Fig. 13 is a flowchart showing the data collection process of the signal processing device 10. Fig. 14A is a flowchart showing the learning process of the signal processing device 10. Fig. 14B is a flowchart showing a subroutine of the learning process. In the learning mode, the signal processing device 10 performs the data collection process of Fig. 13, and then performs the learning process of Figs. 14A and 14B. Fig. 15A is a flowchart showing the threshold calculation process of the signal processing device 10. In the threshold calculation mode, the signal processing device 10 performs the data collection process of Fig. 13, and then performs the threshold calculation process of Fig. 15A. Fig. 15B is a flowchart showing the threshold calculation process. In the case where the separation process is performed in the separation mode and the sensory test process is performed in the sensory test mode, the signal processing device 10 performs the threshold calculation process of Fig. 15B. Fig. 16 is a flowchart showing the judgment process of the signal processing device 10. In the judgment mode, the signal processing device 10 performs the data collection process of Fig. 13, and then performs the judgment process of Fig. 16. However, the signal processing device 10 may perform the judgment process of Fig. 16 in real time while performing the data collection process of Fig. 13. Fig. 17 is a flowchart showing the separation process of the signal processing device 10. In the separation mode, the signal processing device 10 performs the data collection process of Fig. 13, and then performs the separation process of Fig. 17. Fig. 18 is a flowchart showing the sensory test process of the signal processing device 10. In the sensory test mode, the signal processing device 10 performs the sensory test process of Fig. 18 before performing the threshold calculation process of Fig. 15B.

[0112] (Data collection and processing) The data collection process will now be described with reference to FIG. 13, in step S20, the data collecting device 6 inputs the sound pressure signal, the in-cylinder pressure signal (teacher signal), and the angle information to the signal extracting unit 11. In addition, in the case of the separation mode, the threshold calculation mode, or the determination mode, it is not necessary to input the in-cylinder pressure signal in step S20. In step S21, the signal extraction unit 11 calculates the combustion stroke timing of each cylinder from the angle information.

[0113] In step S22, the signal extracting unit 11 extracts the sound pressure signal and the in-cylinder pressure signal near the TDC in accordance with the combustion stroke timing calculated in step S21. In step S23, the spectrogram calculation unit 12 performs a short-time Fourier transform (STFT) on the sound pressure signal extracted in step S22 to calculate a spectrogram of the sound pressure signal. In step S24, the spectrogram calculation unit 12 writes the spectrogram of the sound pressure signal in the signal storage unit 13. Furthermore, the signal extraction unit 11 writes the in-cylinder pressure signal (teacher signal) in the signal storage unit 13 if an in-cylinder pressure signal is present.

[0114] (Learning process) The learning process executed in the learning mode will be described with reference to FIGS. 14A and 14B. As shown in FIG. 14A, in step S30, the signal processing device 10 reads out the spectrogram of the sound pressure signal and the in-cylinder pressure signal from the signal storage unit 13, and inputs them to the learning unit 21. In step S31, the learning unit 21 initializes the weight W of the neural network with a random number in advance, and learns the weight W of the neural network that generates the mask α by taking into account the amplitude and phase components related to the input physical quantity in the neural network 94, using the spectrogram of the sound pressure signal and the in-cylinder pressure signal input in step S30. The learning unit 21 learns the weight W of the neural network that generates the mask α by a convolutional neural network (e.g., U-Net).

[0115] In step S32, the learning unit 21 writes the weight W of the neural network that generates the learned mask α into the learned parameter storage unit 22.

[0116] The process of step S31 is performed by, for example, a series of processes shown in FIG. 14B. As shown in FIG. 14B, in step S31a, the learning processing unit 20 generates a mask α by the neural network 94. In step S101, the learning unit 21 separates the input physical quantity into a noise component and an extracted signal by using the mask α generated in step S31a.

[0117] In step S103, the learning unit 21 updates the weight W of the neural network that generates the mask α. In step S103, the learning unit 21 updates the weight W of the neural network that generates the mask α so that the coherence between the first signal Sfi (FIG. 3A) acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced, and the coherence between the second signal Sse (FIG. 3A) acquired by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted signal (extracted knocking sound 91a) and the teacher signal (observed knocking in-cylinder pressure 93) is increased.

[0118] In step S104, the learning unit 21 determines whether the coherence has converged, and if it is determined that the coherence has converged (in the case of "Yes"), the process of step S31 is terminated, whereas if it is determined that the coherence has not converged (in the case of "No"), the process of step S31a and thereafter is repeated. Note that in step S104, the state in which the coherence has converged is a state in which the coherence between the first signal Sfi (FIG. 3A) and the teacher signal (observed knocking in-cylinder pressure 93) has converged to a value close to "0", and the coherence between the second signal Sse (FIG. 3A) and the teacher signal (observed knocking in-cylinder pressure 93) has converged to a value close to "1".

[0119] (Threshold calculation process) The threshold calculation process executed in the threshold calculation mode will be described with reference to FIG. 15A. As shown in FIG. 15A, in step S40, the learning processing unit 20 generates a mask α by the neural network 94. In step S41, the first estimation unit 23 uses the mask α generated by the neural network 94 to extract the knocking sound of the engine 1 from the spectrogram of the sound pressure signal. In step S42, the threshold calculation unit 24 sums up the absolute values ​​of the spectrograms of the knocking sound in each cycle of the engine 1.

[0120] In step S43, the threshold calculation unit 24 calculates the median value of the spectrogram of the knocking sound in all cycles of the engine 1. In step S44, the threshold calculation unit 24 adds an arbitrary margin to the median value to obtain a threshold value. In step S45, the threshold calculation unit 24 writes the calculated threshold into the threshold storage unit 25.

[0121] The signal processing device 10 has a function of executing the threshold calculation process of Fig. 15B after the separation process of Fig. 17 and the sensory test process of Fig. 18. Hereinafter, the threshold calculation process performed after the separation process of Fig. 17 and the sensory test process of Fig. 18 will be described with reference to Fig. 15B. As shown in FIG. 15B, in step S140, the signal adjustment unit 51 acquires signals that are unacceptable in the sensory test from the noise component storage unit 26c and the extracted signal storage unit 26d, and supplies them to the first estimating unit 23. In step S141, the learning processing unit 20 generates a mask α by the neural network 94. In step S142, the first estimation unit 23 uses the mask α generated by the neural network 94 to obtain a spectrogram of the knocking sound from the spectrogram of the signal that is unacceptable in the sensory test, and supplies the spectrogram to the threshold calculation unit 24. In step S143, the threshold calculation unit 24 sums up the absolute values ​​of the spectrogram of the knocking sound supplied from the first estimating unit 23.

[0122] In step S144, the threshold calculation unit 24 sets an arbitrary value equal to or smaller than the total sum value as the threshold. In step S145, the threshold calculation unit 24 writes the calculated threshold into the threshold storage unit 25.

[0123] (Decision making process) The determination process executed in the determination mode will be described with reference to FIG. As shown in FIG. 16, in step S50, the learning processing unit 20 generates a mask α by the neural network 94. In step S51, the second estimation unit 31 uses the mask α generated by the neural network 94 to extract the knocking sound of the engine 1 from the spectrogram of the sound pressure signal input from the signal storage unit 13.

[0124] In step S52, the threshold determination unit 32 sums up the absolute values ​​of the spectrograms of the knocking sound in each cycle of the engine 1. In step S53, the threshold determination unit 32 compares the sum of the knocking noise with the threshold value already stored in the threshold value storage unit 25, and determines whether the knocking noise exceeds the threshold value. If the sum of the knocking noises exceeds the threshold value (Yes in step S53), the threshold value determining unit 32 determines that knocking has occurred (step S54).

[0125] If the sum of the knocking noises is equal to or smaller than the threshold value (No in step S53), the threshold value determination unit 32 determines that there is no knocking (step S55). In step S56, the threshold determination unit 32 outputs the result of the threshold determination and the knocking sound to the monitor 7.

[0126] (Separation processing) The separation process performed in the separation mode will now be described with reference to FIG. As shown in FIG. 17, in step S60, the learning processing unit 20 generates a mask α by the neural network 94. In step S61, the separator 40 uses the mask α generated by the neural network 94 to separate the input physical quantity into a noise component and an extracted signal. In step S62, the learning processing unit 20 stores the separated noise components and extracted signals in the corresponding noise component storage unit 26c and extracted signal storage unit 26d, respectively.

[0127] (Sensory test treatment) The sensory test process executed in the sensory test mode will be described with reference to FIG. As shown in FIG. 18, the signal adjustment unit 51 of the signal synthesis unit 50 receives the input of level designation information from the level designation unit 9 in step S70. In step S71, the signal adjustment unit 51 acquires the extraction signal from the extraction signal storage unit 26d and the noise component from the noise component storage unit 26c, and changes the level of the extraction signal in accordance with the level designation information to generate a level-changed extraction signal.

[0128] In step S72, the signal adjustment unit 51 synthesizes the level-changed extraction signal and the noise component to generate a processed sound. In step S73, the signal output unit 52 of the signal synthesis unit 50 outputs the processed sound to the sound emitting unit (headphones 8) to cause the sound emitting unit to emit the processed sound. In step S74, the signal synthesis unit 50 determines whether or not an instruction to end has been given from the level designation unit 9, and if it is determined that an instruction to end has been given (if "Yes"), it outputs the processed sound to the first estimation unit 23, calculates a threshold value by the threshold calculation unit 24, writes it into the threshold storage unit 25, and ends the processing of Fig. 18. On the other hand, if it is determined that an instruction to end has not been given (if "No"), in step S75, the signal adjustment unit 51 of the signal synthesis unit 50 accepts a change in the level designation information from the level designation unit 9, and repeats the processing from step S71 onwards.

[0129] After the separation process in FIG. 17 and the sensory test process in FIG. 18, the signal processing device 10 performs a threshold calculation process shown in FIG. 15B.

[0130] In this embodiment, the case where coherence is used as an element representing the relevance has been described, but coherent output power may be used as an element representing the relevance.

[0131] <Main features of the signal processing device> (1) As shown in FIG. 2 and FIG. 3B, the signal processing device 10 according to the present embodiment includes a learning unit 21 and a separation unit 40. As shown in FIG. 3A, the learning unit 21 learns the weight W of a neural network that generates a mask α for removing a noise component (noise 91b) from an input physical quantity (near-engine sound 90) containing the noise component. The separation unit 40 separates the input physical quantity (near-engine sound 90) into the noise component (noise 91b) and an extracted signal (extracted knocking sound 91a) from which the noise component (noise 91b) has been removed, using the mask α. In this configuration, as shown in FIG. 3A, the learning unit 21 learns the weight W of the neural network so that the correlation between the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) becomes small and the correlation between the extracted signal (extracted knocking sound 91a) and the teacher signal (observed knocking in-cylinder pressure 93) becomes large.

[0132] 3A, the signal processing device 10 according to this embodiment uses the relationship between the noise component and the teacher signal (the observed knocking in-cylinder pressure 93) and the relationship between the extracted signal and the teacher signal (the observed knocking in-cylinder pressure 93) for learning, thereby eliminating the need to learn the transfer function portion and shortening the learning time accordingly. Therefore, the weight W of the neural network can be well learned in a short time.

[0133] Moreover, the signal processing device 10 according to the present embodiment can satisfactorily separate a noise component and an extracted signal from a target sound (input physical quantity) for which it is difficult to prepare a desired clean signal in advance. In particular, the signal processing device 10 can separate a noise component and a signal suitable for changing the level in a sensory test. Such a signal processing device 10 can easily evaluate whether or not a target sound (knocking sound) is mixed into a background sound. Therefore, the signal processing device 10 can improve the evaluation performance of the presence or absence of knocking compared to the conventional techniques described in, for example, Patent Documents 2 and 3. Furthermore, the signal processing device 10 can perform a good sensory test, and can make a judgment close to that of an inspector by determining a threshold value based on data that is unacceptable in the sensory test.

[0134] (2) As shown in FIG. 3A, the learning unit 21 of the signal processing device 10 according to the present embodiment may use the coherence between the noise component and the teacher signal (observed knocking in-cylinder pressure 93) as an element representing the association between the noise component and the teacher signal (observed knocking in-cylinder pressure 93). In addition, the learning unit 21 of the signal processing device 10 according to the present embodiment may use the coherence between the extraction signal and the teacher signal (observed knocking in-cylinder pressure 93) as an element representing the association between the extraction signal and the teacher signal (observed knocking in-cylinder pressure 93). Specifically, as shown in FIG. 3A, the learning unit 21 of the signal processing device 10 according to the present embodiment may learn the weight W of the neural network so that the coherence (or coherent output power) between the noise component and the teacher signal (observed knocking in-cylinder pressure 93) is reduced and the coherence (or coherent output power) between the extraction signal and the teacher signal (observed knocking in-cylinder pressure 93) is increased.

[0135] The signal processing device 10 according to the present embodiment does not use a transfer function, but adopts coherence as a loss function, thereby eliminating the need to learn the transfer function and shortening the learning time accordingly. Also, the sound caused by the knocking in-cylinder pressure can be effectively eliminated from the noise 91b (noise component).

[0136] (3) As shown in FIG. 3A, when the learning unit 21 of the signal processing device 10 according to this embodiment learns the weight W of the neural network, it is preferable to perform the learning taking into account the amplitude and phase components related to the input physical quantity (near-engine sound 90).

[0137] The signal processing device 10 according to this embodiment can separate the engine vicinity sound 90 (input physical quantity) into an extracted knocking sound 91a (extracted signal) and noise 91b (noise component) so that the knocking sound (target sound) is not mixed into the noise. Such a signal processing device 10 can improve the evaluation performance of the presence or absence of knocking compared to the conventional techniques described in Patent Documents 2 and 3. Moreover, such a signal processing device 10 can perform a good sensory test.

[0138] (4) As shown in FIG. 7B, the signal processing device 10 according to this embodiment may include a signal synthesis unit 50 that changes and synthesizes the level of either or both of the extracted signal (extracted knocking sound 91a) and the noise component (noise 91b) separated from the input physical quantity (the sound near the engine 90) to generate a processed sound.

[0139] The signal processing device 10 according to this embodiment can make the target sound (knocking sound) to be inspected easier to hear. The signal processing device 10 can allow the inspector to grasp the presence or absence of the target sound (knocking sound) with high accuracy, and can determine a threshold value based on data that is unacceptable in the sensory test, thereby enabling a judgment close to that of the inspector.

[0140] (5) As shown in FIG. 2, the signal synthesis unit 50 of the signal processing device 10 according to this embodiment has a signal adjustment unit 51 that receives a designation of the level of the extracted signal (extracted knocking sound 91a) by the examiner or operator.

[0141] The signal processing device 10 according to this embodiment can arbitrarily and minutely change the level of the extracted signal (extracted knocking sound 91a).

[0142] (6) As shown in Fig. 14A and Fig. 17, the signal processing device 10 according to the present embodiment can realize the following signal processing method. That is, the signal processing method according to the present embodiment includes a learning process (steps S30 to S32 in Fig. 14A) and a separation process (steps S61 to S62 in Fig. 17). In the learning process, a weight W of a neural network that generates a mask α for removing a noise component (noise 91b) from an input physical quantity (near-engine sound 90) containing the noise component (noise 91b) is learned. At that time, the weight W of the neural network is learned so that the correlation between the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced and the correlation between the extracted signal (extracted knocking sound 91a) and the teacher signal (observed knocking in-cylinder pressure 93) is increased. At that time, it is preferable to learn a weight W of a neural network that generates a mask α for removing a noise component (noise 91b) by adding a phase from an input physical quantity (near-engine sound 90) containing the noise component by using the neural network 94. In the separation process, the mask α is used to separate the input physical quantity (near-engine sound 90) into the noise component (noise 91b) and an extracted signal (extracted knocking sound 91a) from which the noise component (noise 91b) has been removed.

[0143] The signal processing method according to the present embodiment can effectively separate an input physical quantity into a noise component and an extracted signal, particularly into a noise component and a signal suitable for level change in a sensory test.

[0144] As described above, according to the signal processing device 10 of the present embodiment, by using the relationship between the first signal acquired from the noise component and the teacher signal, and the relationship between the second signal acquired from the extracted signal and the teacher signal for learning, it is possible to eliminate the learning of the transfer function portion and shorten the learning time accordingly. Also, it is possible to satisfactorily separate the noise component and the extracted signal from the target sound (input physical quantity) for which it is difficult to prepare a desired clean signal in advance. Furthermore, according to the signal processing device 10 of this embodiment, by learning the weights of the neural network so that the coherence between the first signal Sfi (FIG. 3A) obtained by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the noise component (noise 91b) and the teacher signal (observed knocking in-cylinder pressure 93) is reduced, and the coherence between the second signal Sse (FIG. 3A) obtained by performing an inverse short-time Fourier transform (ISTFT) on the spectrogram of the extracted signal (extracted knocking sound 91a) and the teacher signal (observed knocking in-cylinder pressure 93) is increased, it is possible to adopt coherence in the loss function without using a transfer function, thereby shortening the learning time. Moreover, the weight W of the neural network can be learned well so that the processing time is shortened without repeating unnecessary learning processes. Moreover, according to the signal processing device 10 of this embodiment, the input physical quantity can be separated well into a noise component and an extracted signal. In particular, the signal processing device 10 can separate the noise component and a signal suitable for level change in a sensory test. Such a signal processing device 10 can easily evaluate whether or not a target sound (knocking sound) is mixed into the background sound. Therefore, the signal processing device 10 can improve the evaluation performance of the presence or absence of knocking. Moreover, the signal processing device 10 can perform a good sensory test.

[0145] The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the gist of the present invention.

[0146] For example, the above-described embodiment has been described in detail to clearly explain the gist of the present invention. Therefore, the present invention is not necessarily limited to having all of the components described. In addition, the present invention can be modified by adding other components to certain components, or by replacing some components with other components. In addition, the present invention can be modified by deleting some components.

[0147] For example, as described below, the signal processing device 10 can change the test object to an arbitrary article, and can change the input physical quantity and the teacher signal to an arbitrary signal.

[0148] (1) For example, the signal processing device 10 can be used in an environment shown in Fig. 19 after being modified as in a first modified example shown in Fig. 20. Fig. 19 is an explanatory diagram of the first modified example. Fig. 20 is an explanatory diagram of learning the weight W of the neural network that generates a mask in the first modified example.

[0149] FIG. 19 shows an example of evaluating vibration sound of an instrument panel 3b (instrument panel) detected at the ear position of a listener (a passenger in the vehicle (particularly the driver)) in a vehicle interior 3a of a vehicle 3. In the example shown in FIG. 19, a listener's ear position running sound 190 (a vehicle interior sound heard at the listener's ear position) is an input physical quantity, and an observed instrument panel vibration acceleration 193 is a teacher signal. As shown in FIG. 20, the learning unit 21 of the signal processing device 10 separates the input physical quantity (listener's ear position running sound 190) into an extracted signal (extracted instrument panel vibration sound 191a) and a noise component (noise 191b) using a mask α. The mask α includes amplitude and phase components related to the input physical quantity.

[0150] 20, the learning unit 21 of the signal processing device 10 learns the weight W of the neural network so that the coherence between the noise component (noise 191b) and the teacher signal (observed instrument panel vibration acceleration 193) is reduced and the coherence between the extracted signal (extracted instrument panel vibration sound 191a) and the teacher signal (observed instrument panel vibration acceleration 193) is increased. This enables the signal processing device 10 to separate the input physical quantity (listener's ear position running sound 190) into the extracted signal (extracted instrument panel vibration sound 191a) and the noise component (noise 191b) so that the target sound (sound caused by the vibration of the instrument panel 3b) is not mixed into the noise (sound not caused by the vibration of the instrument panel 3b (background sound)).

[0151] Such a signal processing device 10 can make it easier for an inspector to grasp the loudness of the sound caused by the vibration of the instrument panel 3b among the listener's ear position running sound 190 (sound inside the vehicle that can be heard at the listener's ear position) during a sensory test, for example. For example, when the sound caused by the vibration of the instrument panel 3b is relatively loud, the signal processing device 10 can set a target value for vibration reduction of the instrument panel 3b by generating a processed sound in the signal adjustment unit 51 of the signal synthesis unit 50 and subjectively evaluating the sound based on the auditory sense of the inspector.

[0152] (2) The signal processing device 10 can be used in the environment shown in Fig. 21 after being modified as in the second modified example shown in Fig. 22. Fig. 21 is an explanatory diagram of the second modified example. Fig. 22 is an explanatory diagram of learning of weight W of a neural network that generates a mask in the second modified example.

[0153] Fig. 21 shows an example of evaluating compressor vibration noise in the vicinity of refrigerator 203. In the example shown in Fig. 21, refrigerator sound 290 heard at the position of the listener's ears is the input physical quantity, and observed compressor vibration acceleration 293 is the teacher signal. As shown in Fig. 22, learning unit 21 of signal processing device 10 separates the input physical quantity (refrigerator sound 290) into an extracted signal (extracted compressor sound 291a) and a noise component (noise 291b) using mask α. Note that mask α includes amplitude and phase components related to the input physical quantity.

[0154] 22, the learning unit 21 of the signal processing device 10 learns the weight W of the neural network so that the coherence between the noise component (noise 291b) and the teacher signal (observed compressor vibration acceleration 293) is reduced and the coherence between the extracted signal (extracted compressor sound 291a) and the teacher signal (observed compressor vibration acceleration 293) is increased. This enables the signal processing device 10 to separate the input physical quantity (refrigerator sound 290) into the extracted signal (extracted compressor sound 291a) and the noise component (noise 291b) so that the target sound (sound caused by compressor vibration) is not mixed into the noise (sound not caused by compressor vibration (background sound)).

[0155] Such signal processing device 10 can make it easier for an examiner to grasp the loudness of the sound caused by compressor vibration among refrigerator sounds 290 heard at the position of the listener's ears during a sensory test, for example. For example, when the sound caused by compressor vibration is relatively loud, signal processing device 10 can set a target value for reducing compressor vibration by generating processed sound in signal adjustment unit 51 of signal synthesis unit 50 and performing subjective evaluation by the examiner's hearing.

[0156] (3) The signal processing device 10 can be used after being modified as in a third modified example shown in Fig. 23. Fig. 23 is an explanatory diagram of learning of weights W of a neural network that generates a mask in the third modified example.

[0157] The example shown in Fig. 23 shows an example of evaluating combustion noise contained in radiated sound from the engine 1 (Fig. 1) detected at the ear position of a listener (a vehicle occupant (particularly the driver)) in the cabin 3a of the vehicle 3. The example shown in Fig. 23 shows a configuration in which in-cylinder pressure 390 as a teacher signal of the engine 1 (Fig. 1) and radiated sound 390a and vibration 390b as input physical quantities are observed, and masks α1 and α2 are learned based on the respective signals.

[0158] The signal processing device 10 acquires a combustion sound 391aa (extracted signal) from a radiated sound 390a (input physical quantity) using a mask α1, and also acquires a noise 391ab (noise component). The noise 391ab (noise component) is acquired by removing the combustion sound 391aa (extracted signal) from the radiated sound 390a (input physical quantity).

[0159] The learning unit 21 of the signal processing device 10 learns a weight W1 (not shown) of a neural network that generates a mask α1 using the neural network 94 so that the coherence between the noise component (noise 391ab) and the teacher signal (in-cylinder pressure 390) is reduced and the coherence between the extracted signal (combustion sound 391aa) and the teacher signal (in-cylinder pressure 390) is increased. The mask α1 takes into consideration the proportion and phase component of the combustion sound contained in the radiated sound 390a (input physical quantity).

[0160] Furthermore, the signal processing device 10 acquires the combustion oscillation 391ba (extracted signal) from the oscillation 390b (input physical quantity) using the mask α2, and also acquires the noise oscillation 391bb (noise component). The noise oscillation 391bb (noise component) is acquired by removing the combustion oscillation 391ba (extracted signal) from the oscillation 390b (input physical quantity).

[0161] The learning unit 21 of the signal processing device 10 learns a weight W2 (not shown) of a neural network that generates a mask α2 using the neural network 94 so that the coherence between the noise component (noise vibration 391bb) and the teacher signal (in-cylinder pressure 390) is reduced and the coherence between the extracted signal (combustion vibration 391ba) and the teacher signal (in-cylinder pressure 390) is increased. The mask α2 takes into consideration the proportion and phase component of the combustion vibration contained in the vibration 390b (input physical quantity).

[0162] Such a signal processing device 10 can simulate the combustion sound detected at the ear position of a listener (a passenger in a vehicle (particularly the driver)) during, for example, a sensory test. [Explanation of symbols]

[0163] 1. Engine 2. Engine ECU 3. Vehicle 3a…Vehicle compartment 3b…Instrument panel 4...Sound pressure sensor 5...Cylinder pressure sensor 6. Data collection device 7. Monitor 8...Headphones (sound output section) 9…Level designation section 10...Signal processing device 11...Signal cutting section 12...Spectrogram calculation section 13...Signal storage section 14…Switch 20...Learning processing unit 21…Study Department 22…Learned parameter memory section 23…1st estimation part 24...Threshold calculation unit 25...Threshold memory unit 26a...Teacher signal storage section 26c…Noise component storage section 26d...Extracted signal storage section 30...Determination processing unit 31...second estimation unit (extracted signal estimation unit) 32...Threshold value judgment unit 40…Separation part 50...Signal synthesis section 51...Signal conditioning section 52...Signal output section 81...Knocking vibration 82...Knocking sound 83...Mechanical noise (noise component) 90…Near-engine sound (input physical quantity) 91a…Extracted knocking sound (extracted signal) 91aa…Level change knocking sound (level change extraction signal) 91b…Noise (noise component) 91c…processed sound 93…Observed knocking cylinder pressure (teacher signal) 94…Neural network 94A…Mask generation network 95…U-Net 96…Downward pass (Encoder) 97…Hierarchy 98…Upward path (Decoder) 99…Output 100...Signal processing system 190...Listener's ear position traveling sound (input physical quantity) 191a…Extracted instrument panel vibration sound (extracted signal) 191b…Noise (noise component) 193…Observed instrument panel vibration acceleration (teacher signal) 203…Refrigerator 290…Refrigerator sound (input physical quantity) 291a…Extracted compressor sound (extracted signal) 291b…Noise (noise component) 293…Observed compressor vibration acceleration (teacher signal) 390…Cylinder pressure (teacher signal) 390a…Radiated sound (input physical quantity) 390b…Vibration (input physical quantity) 391aa…Combustion sound (extracted signal) 391ab…Noise (noise components) 391ba…Combustion vibration (extracted signal) 391bb…Noise vibration (noise component) α,α1,α2...Mask F11, F21...Spectrogram F12,F22,F93…Signal waveform F13, F23…Coherence M1…Connection for learning mode M2: Connection for threshold calculation mode M3: Connection for judgment mode M4…Connection for separation mode M5: Connection for sensory test mode W…weight Sfi…1st signal Sse…2nd signal T: Threshold

Claims

1. A signal processing device that eliminates the need to learn a transfer function that converts a knocking sound, which is extracted by removing noise components from an input physical quantity using a mask, into an in-cylinder pressure of an internal combustion engine when knocking occurs, comprising: a learning unit that learns weights of a neural network that generates a mask for removing a noise component from an input physical quantity that includes the noise component; a separation unit that separates the input physical quantity into the noise component and an extracted signal from which the noise component has been removed, using the mask; The learning unit learns weights of the neural network so that a correlation between a first signal obtained by performing an inverse short-time Fourier transform on a spectrogram of the noise component and a teacher signal is reduced, and a correlation between a second signal obtained by performing an inverse short-time Fourier transform on a spectrogram of the extracted signal and the teacher signal is increased.

23. A signal processing device comprising:

2. 2. The signal processing device according to claim 1, The learning unit uses the coherence between the first signal and the teacher signal as an element representing the association between the first signal and the teacher signal.

23. A signal processing device comprising:

3. 3. The signal processing device according to claim 1, The learning unit uses the coherence between the second signal and the teacher signal as an element representing the association between the second signal and the teacher signal.

23. A signal processing device comprising:

4. A signal processing device according to any one of claims 1 to 3, the first signal is noise obtained by excluding a knocking sound from the input physical quantity, The second signal is a knocking sound.

23. A signal processing device comprising:

5. A signal processing device according to any one of claims 1 to 4, The mask is a spatial mask expressed in frequency and time.

23. A signal processing device comprising:

6. 6. The signal processing device according to claim 1, The learning unit learns the weights of the neural network by taking into account amplitude and phase components related to the input physical quantity with respect to the mask.

23. A signal processing device comprising:

7. 7. The signal processing device according to claim 1, The apparatus further includes a signal synthesis unit that generates a processed sound by changing and synthesizing a level of either or both of the extracted signal and the noise component separated from the input physical quantity.

23. A signal processing device comprising:

8. A signal processing method that eliminates the need to learn a transfer function that converts a knocking sound extracted by removing noise components from an input physical quantity using a mask into an in-cylinder pressure of an internal combustion engine when knocking occurs, comprising: a learning step of learning weights of a neural network that generates a mask for removing a noise component from an input physical quantity including the noise component; a separation step of separating the input physical quantity into the noise component and an extracted signal from which the noise component has been removed, using the mask; In the learning step, weights of the neural network are learned so that a correlation between a first signal obtained by performing an inverse short-time Fourier transform on a spectrogram of the noise component and a teacher signal is reduced, and a correlation between a second signal obtained by performing an inverse short-time Fourier transform on a spectrogram of the extracted signal and the teacher signal is increased.

23. A computer-implemented signal processing method comprising:

Images