Abnormal sound identifying device, abnormal sound identifying method, and abnormal sound identifying program

Abstract

To provide an abnormal noise specification device, an abnormal noise specification method and an abnormal noise specification program, which precisely specify an abnormal noise by using artificial intelligence.SOLUTION: An abnormal noise specification device includes an arithmetic apparatus and an output device. The arithmetic apparatus performs the steps of: specifying frequency-time data which indicates a time change of a frequency spectrum of sound recorded in a vehicle; designating a first time range and a second time range in the frequency-time data; inputting the frequency-time data to a learned model, allowing the learned model to specify one or a plurality of abnormal noises occurring in the first time range as a first abnormal noise on the basis of the inputted frequency-time data, and allowing the learned model to specify one or a plurality of abnormal noises occurring in the second time range as a second abnormal noise on the basis of the inputted frequency-time data; and allowing the output device to output a type of the first abnormal noise which is not matched with a type of the second abnormal noise in the first abnormal noise.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The technology disclosed in this specification relates to an abnormal sound identifying device, an abnormal sound identifying method, and an abnormal sound identifying program.

[0002] Patent Document 1 discloses a technique for identifying abnormal sounds from sounds recorded by an image forming device. In this technique, sounds recorded by the image forming device are converted into frequency-time data that indicates changes over time in the frequency spectrum. Furthermore, a fast Fourier transform is performed on the frequency-time data in the time axis direction. A user identifies the type of abnormal sound (for example, the source of the abnormal sound) based on the analysis results of the fast Fourier transform and a database of abnormal sounds that have occurred in the past. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-110919 A Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, in Patent Document 1, a user (i.e., a person) identifies abnormal sounds. In response to this, the inventors of the present application have conducted an experiment in which an artificial intelligence is used to identify abnormal sounds from sounds recorded in a vehicle. In the experiment, there are cases in which the artificial intelligence identifies abnormal sounds that are not actually occurring. This specification proposes a technology for accurately identifying abnormal sounds using artificial intelligence. [Means for solving the problem]

[0005] The first abnormal sound identification device disclosed in this specification includes a calculation device that can access a trained model of artificial intelligence, and an output device. The calculation device executes the steps of: identifying frequency-time data indicating a time change in the frequency spectrum of a sound recorded in a vehicle; specifying a first time range and a second time range different from the first time range in the frequency-time data; inputting the frequency-time data to the trained model, causing the trained model to identify one or more abnormal sounds occurring within the first time range as first abnormal sounds based on the input frequency-time data, and causing the trained model to identify one or more abnormal sounds occurring within the second time range as second abnormal sounds based on the input frequency-time data; and outputting, to the output device, the type of the first abnormal sound that does not match the type of the second abnormal sound among the first abnormal sounds.

[0006] In addition, the above-mentioned "step of identifying frequency-time data indicating a time change in the frequency spectrum of the sound recorded in the vehicle" may be any step as long as the frequency-time data is identified (i.e., recognized) by the arithmetic device. For example, the arithmetic device may calculate the frequency-time data based on the sound recorded in the vehicle, or the frequency-time data calculated outside the arithmetic device may be input to the arithmetic device.

[0007] Furthermore, the "trained model" may be located anywhere as long as it is accessible by the computing device. For example, the trained model may be stored in a storage device inside the abnormal sound identification device, or may be stored in a storage device on a network accessible by the computing device.

[0008] Furthermore, the above-mentioned "first time range" and "second time range" may be specified according to an input from a user, or may be specified autonomously by a computing device according to a predetermined algorithm.

[0009] In this abnormal sound identification device, when the arithmetic device identifies the frequency-time data, it specifies a first time range and a second time range in the frequency-time data. The first time range can be specified as a time range in which an abnormal sound is likely to occur, and the second time range can be specified as a time range in which an abnormal sound is unlikely to occur. Next, the arithmetic device inputs the frequency-time data to a trained model, and causes the trained model to identify one or more abnormal sounds occurring within the first time range as a first abnormal sound, and causes the trained model to identify one or more abnormal sounds occurring within the second time range as a second abnormal sound. Next, the arithmetic device outputs to the output device the type of the first abnormal sound that does not match the type of the second abnormal sound among the first abnormal sounds. Since it is highly likely that an abnormal sound does not occur in the second time range, it is highly likely that the second abnormal sound is an abnormal sound that is not recognized by humans. Therefore, the first abnormal sound that matches the type of the second abnormal sound among the first abnormal sounds is also highly likely to be an abnormal sound that is not recognized by humans. Since the computing device causes the output device to output the type of the first abnormal sound that does not match the type of the second abnormal sound among the first abnormal sounds, the output device can display the abnormal sound that is recognizable by a person. In this way, with this abnormal sound identifying device, the type of abnormal sound contained in the sound recorded in the vehicle can be identified with higher accuracy.

[0010] The second abnormal sound identification device disclosed in this specification includes a calculation device that can access a trained model of artificial intelligence, a storage device that stores correspondence data that associates types of abnormal sounds with vehicle parts, and an output device. The calculation device executes the following steps: identifying frequency-time data indicating a time change in the frequency spectrum of a sound recorded in the vehicle and part operation state data indicating a time change in the operation state of the part of the vehicle while the sound was being recorded; inputting the frequency-time data into the trained model and having the trained model identify an abnormal sound contained in the sound as a specific abnormal sound based on the input frequency-time data; identifying a corresponding part that is the part associated with the specific abnormal sound in the correspondence data, identifying an operation time zone of the corresponding part based on the part operation state data of the corresponding part, and determining whether or not to cause the output device to output the type of the specific abnormal sound in a determination process that includes as at least one determination element whether or not the occurrence time zone of the specific abnormal sound overlaps with the operation time zone of the corresponding part.

[0011] In addition, the above-mentioned "frequency-time data indicating a time change in the frequency spectrum of the sound recorded in the vehicle" may be specified in any manner as long as the frequency-time data is specified (i.e., recognized) by the arithmetic device. For example, the arithmetic device may calculate the frequency-time data based on the sound recorded in the vehicle, or frequency-time data calculated outside the arithmetic device may be input to the arithmetic device.

[0012] Furthermore, the above-mentioned "parts operation state data indicating the time change of the operation state of the parts of the vehicle during the recording of the sound" may be specified in any manner as long as the parts operation state data is specified (i.e., recognized) by the computing device. For example, the parts operation state data may be input to the computing device from the outside.

[0013] Furthermore, the "trained model" may be located anywhere as long as it is accessible by the computing device. For example, the trained model may be stored in a storage device inside the abnormal sound identification device, or may be stored in a storage device on a network accessible by the computing device.

[0014] In this abnormal sound identification device, the arithmetic device inputs frequency-time data into the trained model, thereby causing the trained model to identify an abnormal sound contained in a sound recorded in a vehicle as a specific abnormal sound. Then, the arithmetic device identifies a corresponding part that is associated with the specific abnormal sound by the correspondence data. The arithmetic device identifies an operating time period of the corresponding part based on the part operating state data of the corresponding part. Furthermore, the arithmetic device executes a determination process that includes, as at least one of the determination elements, whether or not the occurrence time period of the specific abnormal sound overlaps with the operating time period of the corresponding part. If the occurrence time period of the specific abnormal sound does not overlap with the operating time period of the corresponding part, it is highly likely that the specific abnormal sound has been erroneously detected by the trained model. Therefore, by determining whether or not to output the type of the specific abnormal sound to the output device by this determination process, the type of abnormal sound occurring can be correctly displayed on the output device. In this way, according to this abnormal sound identification device, the type of abnormal sound contained in a sound recorded in a vehicle can be identified with higher accuracy. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of an abnormal sound identifying device 10 according to a first embodiment. [Diagram 2] FIG. [Diagram 3] 4 is a flowchart showing an abnormal sound identifying method according to the first embodiment. [Figure 4] A diagram showing STFT data 54. [Diagram 5] FIG. 4 is a diagram illustrating a first time range T1 and a second time range T2. [Figure 6] 6 is a flowchart showing a determination process based on a first abnormal sound and a second abnormal sound. [Figure 7] FIG. 13 is a diagram showing a display screen of the abnormal sound identification result. [Figure 8]11 is a flowchart showing a determination process according to a modified example of the first embodiment. [Figure 9] 11 is a flowchart showing a determination process according to a modified example of the first embodiment. [Figure 10] FIG. 11 is a block diagram of an abnormal sound identifying device 10 according to a second embodiment. [Figure 11] FIG. [Figure 12] 10 is a flowchart showing an abnormal sound identifying method according to a second embodiment. [Figure 13] 6 is a flowchart showing a determination process based on a time period during which an abnormal noise occurs and a time period during which a corresponding part is in operation. [Figure 14] FIG. 13 is a diagram illustrating an example of STFT data 154 and part operating state data 156b. [Figure 15] FIG. 13 is a diagram illustrating an example of STFT data 154 and part operating state data 156b. [Figure 16] FIG. 13 is a diagram illustrating an example of STFT data 154 and part operating state data 156b. [Figure 17] FIG. 13 is a diagram illustrating an example of STFT data 154 and part operating state data 156b. [Figure 18] 13 is a flowchart showing a determination process according to a modified example of the second embodiment. [Figure 19] 13 is a flowchart showing a determination process according to a modified example of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In one example of the first abnormal sound identification device disclosed in the present specification, in the step of causing the output device to output, the calculation device may not cause the output device to output the type of the first abnormal sound that matches the type of the second abnormal sound among the first abnormal sounds.

[0017] In one example of the first abnormal sound identifying device disclosed in the present specification, a sound pressure level of the sound within the first time range may be higher than a sound pressure level of the sound within the second time range. EXAMPLES

[0018] The abnormal sound identification device 10 shown in FIG. 1 identifies an abnormal sound from a sound recorded in a vehicle. The abnormal sound identification device 10 is configured by a so-called computer. The abnormal sound identification device 10 has a calculation device 12, a storage device 16, a monitor 18, a speaker 20, an input device 22, a network interface 24, and the like. The calculation device 12 is configured by a CPU (central processing unit), a memory, and the like. The calculation device 12 is connected to the storage device 16, the monitor 18, the speaker 20, the input device 22, and the network interface 24. The storage device 16 is configured by a hard disk drive, a solid state drive, and the like. The storage device 16 stores an abnormal sound identification program 50. The calculation device 12 executes the abnormal sound identification program 50. The input device 22 is configured by a mouse, a keyboard, and the like. A signal is input to the calculation device 12 by a user operating the input device 22. The calculation device 12 controls the monitor 18 and the speaker 20. The arithmetic device 12 is also connected to a network line 30 via a network interface 24. The network line 30 may be the Internet or an intranet. A storage device 32 and the like are connected to the network line 30. At least one of the storage device 16 and the storage device 32 stores a trained model 52. Regardless of whether the trained model 52 is stored in the storage device 16 or the storage device 32, the arithmetic device 12 can access the trained model 52. The trained model 52 is an artificial intelligence and is configured by a convolutional network (CNN). The trained model 52 is a model that has been trained to identify abnormal sounds from frequency-time data (data indicating time changes in the frequency profile of a sound). In this embodiment, short-time Fourier transform data (hereinafter referred to as STFT data) is used as the frequency-time data. When STFT data is input to the trained model 52, the trained model 52 identifies the type of abnormal sound contained in the sound based on the STFT data.

[0019] The storage device 16 can store sound data 56. The sound data 56 is generated by the sound recording device 42 shown in FIG. 2. The sound recording device 42 is mounted on the vehicle 40. The sound recording device 42 records sounds generated from the vehicle 40 while the vehicle 40 is running. The sound recording device 42 may record inside the vehicle cabin or outside the vehicle cabin (for example, inside the engine room, etc.). The sound recording device 42 saves the recorded sounds as sound data 56 in a portable storage device. The sound data 56 is data indicating a waveform of sound vibrations (i.e., air vibrations). The sound data 56 includes vehicle road noise, engine noise, and operating sounds of parts (for example, VSV (vacuum switching valve), alternator, water pump, turbo, etc.). In addition, if an abnormal sound is generated in the vehicle 40 for some reason, the abnormal sound is also included in the sound data 56. By connecting a portable storage device to the abnormal sound identifying device 10, the sound data 56 can be input to the abnormal sound identifying device 10. Also, sound data 56 can be input to the abnormal sound identifying device 10 via the network line 30. The sound data 56 input to the abnormal sound identifying device 10 is stored in the storage device 16, as shown in FIG.

[0020] Next, a description will be given of an abnormal sound identification method executed by the abnormal sound identification device 10. The abnormal sound identification device 10 executes an abnormal sound identification program 50 to execute the abnormal sound identification method shown in Fig. 3. When a user performs a predetermined operation using the input device 22, the arithmetic device 12 starts the abnormal sound identification program 50 (i.e., the abnormal sound identification method in Fig. 3).

[0021] In step S2, the arithmetic device 12 instructs the user to select sound data 56 by displaying a window or the like on the monitor 18. The user can select any sound data 56 from the sound data 56 stored in the storage device 16 by using the input device 22.

[0022] In step S4, the arithmetic device 12 performs a short-time Fourier transform on the sound data 56 selected in step S2. As a result, the arithmetic device 12 calculates the STFT data 54 shown in Fig. 4. The vertical axis of the STFT data 54 indicates the frequency of the sound, and the horizontal axis of the STFT data 54 indicates time. Furthermore, the color of each pixel in the STFT data 54 indicates the sound pressure level (dB). In other words, the STFT data 54 is data that indicates the change over time in the frequency spectrum of the sound. The arithmetic device 12 calculates the STFT data 54 as image data.

[0023] In step S6, the arithmetic device 12 displays the STFT data 54 on the monitor 18 as shown in FIG. 5, and instructs the user to select a first time range T1 and a second time range T2. The user can select the first time range T1 and the second time range T2 from the time range shown on the horizontal axis of the STFT data 54 by the input device 22. The arithmetic device 12 instructs the user to select a time range in which an abnormal sound is considered to be included as the first time range T1. The arithmetic device 12 also instructs the user to select a time range in which an abnormal sound is not considered to be included as the second time range T2. The user can select the first time range T1 and the second time range T2 while visually checking the STFT data 54. For example, the user can select a time range in which the sound pressure level (i.e., the sound pressure level in all frequency bands) is high as the first time range T1, and select a time range in which the sound pressure level is lower than the first time range T1 as the second time range T2. As shown in FIG. 5, a play button 90 and a confirm button 92 are displayed on the monitor 18 next to the STFT data 54. When the play button 90 is pressed with a time range selected, the arithmetic device 12 plays the sound within the selected time range through the speaker 20. Therefore, the user can determine whether or not an abnormal sound is included within the selected time range by listening to the sound within the selected time range. This allows the user to select a time range that includes an abnormal sound as the first time range T1 and a time range that does not include an abnormal sound as the second time range T2. In addition, when an abnormal sound to be detected exists, the user can select a time range in which the abnormal sound to be detected exists as the first time range T1 and a time range in which the abnormal sound does not exist as the second time range T2. When the user presses the confirm button 92 with the first time range T1 and the second time range T2 selected, the arithmetic device 12 confirms the selected first time range T1 and second time range T2 and proceeds to the next step.

[0024] In step S8, the calculation device 12 inputs the STFT data 54 calculated in step S4 to the trained model 52. More specifically, the calculation device 12 first inputs the STFT data 54-1 within the first time range T1 to the trained model 52, and then inputs the STFT data 54-2 within the second time range T2 to the trained model 52.

[0025] When the STFT data 54-1 within the first time range T1 is input to the trained model 52, the trained model 52 extracts features from the input STFT data 54-1 and identifies an abnormal sound included in the STFT data 54-1 (more specifically, an abnormal sound included in the sound represented by the STFT data 54-1) based on the extracted features. That is, the trained model 52 identifies a first abnormal sound occurring within the first time range T1. The trained model 52 identifies the first abnormal sound and, at the same time, identifies the type of the first abnormal sound. The trained model 52 identifies abnormal sound A, abnormal sound B, etc. as the type of abnormal sound. In a more specific example, the trained model 52 identifies VSV abnormal sound, alternator abnormal sound, water pump abnormal sound, turbo abnormal sound, etc. as the abnormal sound. That is, the type of abnormal sound represents the source of the abnormal sound. In step S8, multiple first abnormal sounds may be identified.

[0026] When the STFT data 54-2 within the second time range T2 is input to the trained model 52, the trained model 52 extracts features from the input STFT data 54-2 and identifies an allophone included in the STFT data 54-2 (more specifically, an allophone included in the sound represented by the STFT data 54-2) based on the extracted features. That is, the trained model 52 identifies a second allophone occurring within the second time range T2. The trained model 52 identifies the second allophone and, at the same time, identifies the type of the second allophone. In step S8, multiple second allophones may be identified.

[0027] As described above, in step S8, the first abnormal sound and the second abnormal sound are identified.

[0028] In step S10, the calculation device 12 executes a determination process based on the first abnormal sound and the second abnormal sound, and displays the type of the first abnormal sound on the monitor 18 according to the determination result. Fig. 6 shows details of step S10. As shown in Fig. 6, in step S10, the calculation device 12 executes steps S20 to S28.

[0029] In step S20, the calculation device 12 selects one first abnormal sound from among the first abnormal sounds identified in step S8. If only one abnormal sound is identified in step S8, that first abnormal sound is selected.

[0030] In step S22, the calculation device 12 determines whether the type of the first abnormal sound selected in step S20 matches the type of any of the second abnormal sounds identified in step S8. For example, if the first abnormal sound selected in step S20 is abnormal sound A and the second abnormal sounds identified in step S8 are abnormal sound A and abnormal sound B, the calculation device 12 determines YES in step S22. Also, for example, if the first abnormal sound selected in step S20 is abnormal sound C and the second abnormal sounds identified in step S8 are abnormal sound A and abnormal sound B, the calculation device 12 determines NO in step S22.

[0031] If the determination in step S22 is NO, the calculation device 12 executes step S24. In step S24, the calculation device 12 displays the type of the first abnormal sound selected in step S20 on the monitor 18. For example, if the first abnormal sound selected in step S20 is abnormal sound A, then in step S24, "Abnormal sound A has been detected" is displayed on the monitor 18 as shown in FIG. 7. On the other hand, if the determination in step S22 is YES, the calculation device 12 executes step S26. In step S26, the calculation device 12 does not display the type of the first abnormal sound selected in step S20 on the monitor 18.

[0032] In step S28, the arithmetic device 12 determines whether or not the processing for all the first abnormal sounds identified in step S8 has been completed. If the result is NO in step S28, step S20 is executed again. In this case, in the next step S20, a first abnormal sound for which the processing has not been completed is selected. Therefore, steps S20 to S28 are repeated until the processing for all the first abnormal sounds is completed. Therefore, when a plurality of first abnormal sounds are identified by the trained model 52, only the first abnormal sounds determined to be NO in step S22 among the plurality of first abnormal sounds are displayed on the monitor 18. Note that, when there are a plurality of first abnormal sounds determined to be NO in step S22, these first abnormal sounds are displayed as a list on the monitor 18 in step S24. Also, when there is no first abnormal sound determined to be NO in step S22, the monitor 18 displays "An abnormal sound could not be detected." When the processing for all the first abnormal sounds identified by the trained model 52 is completed, the arithmetic device 12 ends the abnormal sound determination process.

[0033] As described above, the arithmetic device 12 displays the selected type of the first abnormal sound on the monitor 18 when the type of the first abnormal sound being selected does not match the type of the second abnormal sound, and does not display the selected type of the first abnormal sound on the monitor 18 when the type of the first abnormal sound being selected matches the type of the second abnormal sound. This makes it possible to display only the abnormal sounds that the user recognizes as abnormal sounds on the monitor 18. That is, as described above, the first time range T1 is a time range in which the user determines that an abnormal sound is occurring. Therefore, the first abnormal sound identified by the trained model 52 within the first time range T1 includes an abnormal sound recognized by the user. On the other hand, the first abnormal sound identified by the trained model 52 within the first time range T1 may include an abnormal sound that is not recognized by the user. Also, as described above, the second time range T2 is a time range in which the user determines that no abnormal sound is occurring. Therefore, the second abnormal sound identified by the trained model 52 within the second time range T2 is an abnormal sound that is not recognized by the user. Therefore, the first abnormal sound whose type matches the second abnormal sound among the first abnormal sounds is considered to be an abnormal sound not recognized by the user. Therefore, the calculation device 12 displays the first abnormal sound whose type does not match the second abnormal sound among the first abnormal sounds on the monitor 18, and does not display the first abnormal sound whose type matches the second abnormal sound among the first abnormal sounds on the monitor 18. As a result, of the abnormal sounds identified by the trained model 52 within the first time range T1, only the abnormal sounds recognized by the user are displayed on the monitor 18. Therefore, the abnormal sound identifying device 10 can accurately identify the abnormal sound recognized as an abnormal sound by the user.

[0034] As described above, according to the abnormal sound identifying device 10 of the first embodiment, it is possible to select only those abnormal sounds that the user recognizes as abnormal sounds from among the abnormal sounds identified by the trained model 52, and to display the types of those abnormal sounds on the monitor 18. In other words, it is possible to accurately identify only those abnormal sounds that the user actually notices.

[0035] In the first embodiment, the first time range T1 and the second time range T2 are specified by the user, but the first time range T1 and the second time range T2 may be automatically specified by the calculation device 12. For example, the calculation device 12 may specify a time range in which the sound pressure level is high as the first time range T1, and a time range in which the sound pressure level is low as the second time range T2. Even if the first time range T1 and the second time range T2 are automatically specified in this manner, the second abnormal sound identified in the second time range T2 in which the sound pressure level is low is likely to be an abnormal sound that the user cannot recognize. Therefore, by displaying only the first abnormal sound whose type does not match the second abnormal sound on the monitor 18, it is possible to display only the abnormal sounds that the user can recognize on the monitor 18.

[0036] In Fig. 6, if the determination in step S22 is NO, the selected first abnormal sound is displayed on the monitor 18 in step S24. However, as shown in Fig. 8, step S23 in which another determination is made may be present between step S22 and step S24. In Fig. 8, even if the determination in step S22 is NO, if the selected first abnormal sound does not satisfy criterion condition 1 (i.e., if the determination in step S23 is NO), the selected first abnormal sound is not displayed on the monitor 18. In this way, even if the determination in step S22 is NO, there may be cases where the selected first abnormal sound is not displayed on the monitor 18.

[0037] Also, in FIG. 6, if the determination in step S22 is YES, the selected first abnormal sound is not displayed on the monitor 18 in step S26. However, as shown in FIG. 9, there may be step S25, in which another determination is made, between step S22 and step S26. In FIG. 9, even if the determination in step S22 is YES, if the selected first abnormal sound satisfies criterion condition 2 (i.e., if the determination in step S25 is YES), the selected first abnormal sound is displayed on the monitor 18. In this way, even if the determination in step S22 is YES, there may be cases where the selected first abnormal sound is displayed on the monitor 18. Also, FIG. 8 and FIG. 9 may be combined. EXAMPLES

[0038] The abnormal sound identifying device 100 of the second embodiment shown in FIG. 10 identifies an abnormal sound from a sound recorded in a vehicle. The abnormal sound identifying device 100 of the second embodiment has the same hardware configuration as the abnormal sound identifying device 10 of the first embodiment. In the second embodiment, an abnormal sound identifying program 150 is stored in the storage device 16. The abnormal sound identifying program 150 of the second embodiment is different from the abnormal sound identifying program 50 of the first embodiment. In the second embodiment, the storage device 16 stores correspondence data 158. In the second embodiment, vehicle measurement data 156, which is a set of sound data 156a and part operating state data 156b, is input to the storage device 16. The other configurations of the abnormal sound identifying device 100 of the second embodiment are the same as those of the abnormal sound identifying device 10 of the first embodiment.

[0039] The sound data 156a of the second embodiment is data of sound recorded in the vehicle 40 by the recording device 42 shown in FIG. 2, similar to the sound data 56 of the first embodiment. In the second embodiment, the operating state of each part of the vehicle 40 is measured at the same time as the sound data 156a is recorded. For example, the operating states of the VSV, the alternator, the water pump, the turbo, etc. are measured. The data indicating the operating state of the parts measured in the vehicle 40 is the above-mentioned part operating state data 156b. Each part operating state data 156b is measured in synchronization with the sound data 156a. That is, the above-mentioned vehicle measurement data 156 is data configured of a set of the sound data 156a and a plurality of part operating state data 156b. The vehicle measurement data 156 is input to the abnormal sound identifying device 100. The vehicle measurement data 156 input to the abnormal sound identifying device 100 is stored in the storage device 16.

[0040] Correspondence data 158 is data that associates the type of abnormal sound generated in vehicle 40 with the part that is the source of the abnormal sound. For example, in correspondence data 158, as shown in Fig. 11, the VSV is associated with the source of abnormal sound A, the alternator is associated with the source of abnormal sound B, the water pump is associated with the source of abnormal sound C, and the turbo is associated with the source of abnormal sound D.

[0041] Next, a description will be given of an abnormal sound identification method executed by the abnormal sound identification device 100. The abnormal sound identification device 100 executes the abnormal sound identification method shown in Fig. 12 by executing an abnormal sound identification program 150. When a user performs a predetermined operation using the input device 22, the arithmetic device 12 starts the abnormal sound identification program 150 (i.e., the abnormal sound identification method of Fig. 12).

[0042] In step S52, the computing device 12 instructs the user to select vehicle measurement data 156 by displaying a window or the like on the monitor 18. The user can use the input device 22 to select any vehicle measurement data 156 from the vehicle measurement data 156 stored in the storage device 16.

[0043] In step S54, the calculation device 12 performs a short-time Fourier transform on the sound data 156a included in the vehicle measurement data 156 selected in step S52. As a result, the calculation device 12 calculates the STFT data 154 shown in Fig. 14. The vertical axis of the STFT data 154 indicates the frequency of the sound, and the horizontal axis of the STFT data 154 indicates time. Furthermore, the color of each pixel in the STFT data 154 indicates the sound pressure level (dB). In other words, the STFT data 154 is data that indicates the time change of the frequency spectrum of the sound. The calculation device 12 calculates the STFT data 154 as image data.

[0044] 14 also shows part operation state data 156bv of the VSV as an example of the part operation state data 156b. The horizontal axis of the part operation state data 156bv indicates time, and is synchronized with the horizontal axis of the STFT data 154. The vertical axis of the part operation state data 156bv indicates the drive duty (%) of the VSV. A period when the drive duty is high is a period when the VSV is operating, and a period when the drive duty is low is a period when the VSV is stopped. In this way, the calculation device 12 converts the sound data 156a into STFT data 154, thereby obtaining data in which the STFT data 154 and the part operation state data 156b are synchronized.

[0045] In step S56, the arithmetic device 12 inputs the STFT data 154 calculated in step S54 to the trained model 52. The trained model 52 then extracts features from the input STFT data 154 and identifies an abnormal sound included in the STFT data 154 (more specifically, an abnormal sound included in the sound represented by the STFT data 154) based on the extracted features. The trained model 52 identifies an abnormal sound and, at the same time, identifies a type of the abnormal sound. The trained model 52 identifies abnormal sound A, abnormal sound B, abnormal sound C, abnormal sound D, etc. as the type of abnormal sound. In a more specific example, the trained model 52 identifies a VSV abnormal sound, an alternator abnormal sound, a water pump abnormal sound, a turbo abnormal sound, etc. as the abnormal sound. In step S56, the trained model 52 may identify a plurality of types of abnormal sounds.

[0046] In step S58, the arithmetic device 12 identifies a time period T3 during which an abnormal sound occurs. For example, if the trained model 52 has a function called gradient-weighted class activation mapping (Grad-CAM), the trained model 52 outputs a grounds range, which is a frequency range and a time range that are the grounds for identifying an abnormal sound, in step S56. In this case, the arithmetic device 12 can identify a time period during which an abnormal sound occurs from the grounds range output by the trained model 52. For example, if the trained model 52 outputs a grounds range 60 as shown in FIG. 14, the arithmetic device 12 can identify a time range corresponding to the grounds range 60 as the time period T3 during which an abnormal sound occurs. The arithmetic device 12 may also identify a time period during which an abnormal sound occurs by other methods. For example, a time period during which a sound pressure level is high may be identified as the time period during which an abnormal sound occurs. If multiple abnormal sounds are identified in step S56, the arithmetic device 12 identifies a time period T3 during which an abnormal sound occurs for each abnormal sound in step S58.

[0047] In step S60, the arithmetic device 12 reads out the correspondence data 158 from the storage device 16. The arithmetic device 12 identifies the part that is the source of the abnormal sound identified in step S56 based on the correspondence data 158. For example, if an abnormal sound A is identified in step S56, the arithmetic device 12 identifies a VSV as the source of the abnormal sound A based on the correspondence data 158 (i.e., FIG. 11). Hereinafter, the part identified in step S60 (i.e., the part corresponding to the abnormal sound identified in step S56) will be referred to as a corresponding part. Furthermore, if multiple abnormal sounds are identified in step S56, the arithmetic device 12 identifies a corresponding part for each abnormal sound in step S60.

[0048] In step S62, the calculation device 12 identifies the operating time period T4 of the corresponding part that is the source of the abnormal noise. The calculation device 12 reads out the part operating state data 156b of the corresponding part identified in step S60. Then, the calculation device 12 identifies the operating time period of the corresponding part from the part operating state data 156b. For example, if the corresponding part is a VSV, the calculation device 12 reads out the part operating state data 156bv of the VSV shown in FIG. 14. The calculation device 12 sets a predetermined threshold value Dth for the drive duty of the VSV, and identifies a period in which the drive duty is higher than the threshold value Dth as the operating time period T4 of the VSV. In this way, when the part operating state data 156b of the corresponding part is represented by a numerical value such as the drive duty, the calculation device 12 calculates the operating time period T4 by setting a threshold value. Also, when the part operating state data 156b of the corresponding part is represented by a binary value of on-off, the calculation device 12 can identify the on period as the operating time period T4. In this manner, in step S62, the calculation device 12 reads out the part operating state data 156b of the corresponding part, and identifies the operating time zone T4 of the corresponding part from the read out part operating state data 156b. Note that, when there are multiple corresponding parts, the calculation device 12 identifies the operating time zone T4 for each corresponding part.

[0049] In step S64, the calculation device 12 executes a determination process based on the abnormal noise occurrence time period T3 and the corresponding part operation time period T4, and displays the type of abnormal noise on the monitor 18 according to the determination result. Fig. 13 shows the details of step S64. As shown in Fig. 13, in step S64, the calculation device 12 executes steps S70 to S78.

[0050] In step S70, the calculation device 12 selects one abnormal sound from among the abnormal sounds identified in step S56. If only one abnormal sound is identified in step S56, that abnormal sound is selected.

[0051] In step S72, the calculation device 12 judges whether or not the time period T3 during which the abnormal sound is generated selected in step S70 overlaps with the operating time period T4 of the part corresponding to the abnormal sound. For example, if the abnormal sound selected in step S70 is abnormal sound A, the corresponding part is VSV as shown in FIG. 11. Therefore, the calculation device 12 judges whether or not the time period T3 during which the abnormal sound A is generated overlaps with the operating time period T4 of the VSV. If the time period T3 during which the abnormal sound A is generated overlaps with the operating time period T4 of the VSV as shown in FIGS. 14 and 15, the calculation device 12 judges YES in step S72. Also, if the time period T3 during which the abnormal sound A is generated does not overlap with the operating time period T4 of the VSV as shown in FIG. 16, the calculation device 12 judges NO in step S72. Also, if the VSV is not operating at all in the entire time range indicated by the STFT data 154 as shown in FIG. 17, the operating time period T4 of the VSV does not exist. Therefore, the time period T3 during which the abnormal sound A is generated does not overlap with the operating time period T4 of the VSV. Therefore, in this case, the calculation device 12 determines NO in step S72.

[0052] If the determination in step S72 is YES, the calculation device 12 executes step S74. In step S74, the calculation device 12 displays the type of abnormal sound selected in step S70 on the monitor 18. For example, if the selected abnormal sound is abnormal sound A, "Abnormal sound A has been detected" is displayed on the monitor 18 in step S74. On the other hand, if the determination in step S72 is NO, the calculation device 12 executes step S76. In step S76, the calculation device 12 does not display the type of abnormal sound selected in step S70 on the monitor 18.

[0053] In step S78, the arithmetic device 12 judges whether or not the processing for all the abnormal sounds identified by the trained model 52 has been completed. If NO in step S78, step S70 is executed again. In this case, in the next step S70, an abnormal sound for which processing has not been completed is selected. Therefore, steps S70 to S78 are repeated until the processing for all the abnormal sounds is completed. Therefore, when a plurality of abnormal sounds are identified by the trained model 52, only the abnormal sounds determined as YES in step S72 among the plurality of abnormal sounds are displayed on the monitor 18. Note that, when there are a plurality of abnormal sounds determined as YES in step S72, these abnormal sounds are displayed as a list on the monitor 18 in step S74. Also, when there is no abnormal sound determined as YES in step S72, "An abnormal sound could not be detected." is displayed on the monitor 18 in step S74. When the processing for all the abnormal sounds identified by the trained model 52 is completed, the arithmetic device 12 ends the abnormal sound determination processing.

[0054] As described above, the arithmetic device 12 displays the type of abnormal sound being selected on the monitor 18 when the time period T3 during which the abnormal sound is generated overlaps with the time period T4 during which the corresponding part of the abnormal sound is in operation, and does not display the type of abnormal sound being selected on the monitor 18 when the time period T3 during which the abnormal sound is generated overlaps with the time period T4 during which the corresponding part of the abnormal sound is in operation. This makes it possible to accurately display the type of abnormal sound that is actually occurring on the monitor 18. That is, the trained model 52 may identify an abnormal sound that is not actually occurring. The abnormal sound identification device 100 of the second embodiment extracts, from the abnormal sounds identified by the trained model 52, an abnormal sound whose time period T3 during which the abnormal sound is generated overlaps with the time period T4 during which the corresponding part of the abnormal sound is in operation, and displays the extracted abnormal sound on the monitor 18. This makes it possible to prevent an abnormal sound generated by a corresponding part from being erroneously displayed on the monitor 18 even when the corresponding part is not in operation. Therefore, this abnormal sound identification device 100 makes it possible to accurately display the type of abnormal sound that is actually occurring.

[0055] In FIG. 13, if the determination in step S72 is YES, the selected abnormal sound is displayed on the monitor 18 in step S74. However, as shown in FIG. 18, there may be step S73 in which another determination is made between steps S72 and S74. In FIG. 18, even if the determination in step S72 is YES, if the selected abnormal sound does not satisfy criterion condition 3 (i.e., if the determination in step S73 is NO), the selected abnormal sound is not displayed on the monitor 18. In this way, even if the determination in step S72 is YES, there may be cases where the selected abnormal sound is not displayed on the monitor 18.

[0056] Also, in FIG. 13, if the determination in step S72 was NO, the selected abnormal sound was not displayed on monitor 18 in step S76. However, as shown in FIG. 19, step S75 in which another determination is made may be present between step S72 and step S76. In FIG. 19, even if the determination in step S72 is NO, if the selected abnormal sound satisfies criterion condition 4 (i.e., YES in step S75), the selected abnormal sound is displayed on monitor 18. In this way, even if the determination in step S72 is NO, there may be cases where the selected abnormal sound is displayed on monitor 18. Also, FIG. 18 and FIG. 19 may be combined.

[0057] In the above-described first and second embodiments, the arithmetic device 12 displays the type of abnormal sound on the monitor 18. However, the arithmetic device 12 may output the type of abnormal sound to another device. For example, the arithmetic device 12 may output the type of abnormal sound to another device via the network line 30.

[0058] In addition, in the above-described first and second embodiments, the STFT data 54, 154 was image data. That is, the trained model 52 identified the type of abnormal sound based on the STFT data 54, 154, which is image data. However, the data format of the STFT data 54, 154 is not limited to image data. For example, the STFT data 54, 154 may be data indicating a numerical value of the sound pressure level for each time and frequency.

[0059] Although the embodiments have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and changes to the specific examples exemplified above. The technical elements described in this specification or drawings exhibit technical utility alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the technology exemplified in this specification or drawings achieves multiple objectives simultaneously, and achieving one of the objectives itself has technical utility. [Explanation of symbols]

[0060] 10: Abnormal sound identification device 12: Arithmetic device 16:Storage device 18: Monitor 20: Speaker 22: Input device 24: Network interface 30: Network line 32: Storage device

Claims

1. An abnormal sound identifying device, A computing device capable of accessing a trained model of artificial intelligence; An output device; having The computing device, determining frequency-time data indicative of the time evolution of the frequency spectrum of sounds recorded from the vehicle; having a user specify a first time range and a second time range different from the first time range in the frequency-time data; inputting the specified frequency-time data into the trained model, and causing the trained model to identify one or more allophane sounds occurring within the first time range as first allophane sounds based on the input frequency-time data, and causing the trained model to identify one or more allophane sounds occurring within the second time range as second allophane sounds based on the input frequency-time data; causing the output device to output a type of the first abnormal sound that does not match a type of the second abnormal sound; Execute Abnormal noise identification device.

2. 2. The abnormal sound identifying device according to claim 1, wherein in the step of causing the output device to output, the arithmetic device does not cause the output device to output a type of the first abnormal sound that matches a type of the second abnormal sound among the first abnormal sounds.

3. 3. The abnormal sound identifying device according to claim 1, wherein a sound pressure level of the sound within the first time range is higher than a sound pressure level of the sound within the second time range.

4. An abnormal sound identification device as described in claim 1 or 2, wherein the frequency-time data identified in the identification step indicates a time change in the frequency spectrum of the sound recorded with the vehicle in motion.

5. An abnormal sound identification device as described in any one of claims 1 to 4, wherein, in the step of having the user specify the first time range and the second time range, the computing device has the user specify a time range in which it is determined that an abnormal sound is occurring as the first time range, and a time range in which it is determined that no abnormal sound is occurring as the second time range.

6. An abnormal sound identification method for identifying an abnormal sound from a sound recorded in a vehicle using a calculation device and an output device that can access a learned model of artificial intelligence, comprising: The computing device identifies frequency-time data indicating a time change in a frequency spectrum of the sound; the computing device prompting a user to specify a first time range and a second time range different from the first time range in the frequency-time data; a step in which the arithmetic device inputs the specified frequency-time data into the trained model, and causes the trained model to identify one or more allostatic sounds occurring within the first time range as first allostatic sounds based on the input frequency-time data, and causes the trained model to identify one or more allostatic sounds occurring within the second time range as second allostatic sounds based on the input frequency-time data; a step of causing the arithmetic device to output, to the output device, a type of the first abnormal sound that does not match a type of the second abnormal sound; The abnormal sound identification method according to the present invention comprises the steps of:

7. The method for identifying an abnormal sound as described in claim 6, wherein the frequency-time data identified in the identifying step indicates a time change in the frequency spectrum of the sound recorded with the vehicle in motion.

8. The method for identifying abnormal sounds described in claim 6 or 7, wherein, in the step of having the user specify the first time range and the second time range, the computing device has the user specify a time range in which it is determined that an abnormal sound is occurring as the first time range, and a time range in which it is determined that no abnormal sound is occurring as the second time range.

9. An abnormal sound identification program, The trained model of the artificial intelligence is accessible and executable by a computing device connected to an output device; When the arithmetic device executes the abnormal sound identification program, the arithmetic device determining frequency-time data indicative of the time evolution of the frequency spectrum of sounds recorded from the vehicle; having a user specify a first time range and a second time range different from the first time range in the frequency-time data; inputting the specified frequency-time data into the trained model, and causing the trained model to identify one or more allophane sounds occurring within the first time range as first allophane sounds based on the input frequency-time data, and causing the trained model to identify one or more allophane sounds occurring within the second time range as second allophane sounds based on the input frequency-time data; causing the output device to output a type of the first abnormal sound that does not match a type of the second abnormal sound; An abnormal noise identification program configured to run.

10. The abnormal sound identification program as described in claim 9, wherein the frequency-time data identified in the identifying step indicates a time change in the frequency spectrum of the sound recorded with the vehicle in motion.

11. An abnormal sound identification program as described in claim 9 or 10, wherein, in the step of having the user specify the first time range and the second time range, the computing device has the user specify a time range in which it is determined that an abnormal sound is occurring as the first time range, and a time range in which it is determined that no abnormal sound is occurring as the second time range.

Images