Image processing device and image processing method

Abstract

An image processing device (1) includes a controller (20). The controller (20) calculates a fundamental frequency component and a higher harmonic component corresponding to the fundamental frequency component contained in sound data, converts the fundamental frequency component and the higher harmonic component into image data, and generates a sound image in which the fundamental frequency component and the higher harmonic component converted into image data are arranged adjacent to each other.

Description

Technical Field

[0001] The present invention relates to an image processing device and an image processing method. Background Art

[0002] Currently, there is a known method for determining whether the sound of a device is normal or abnormal (Patent Document 1). The invention described in Patent Document 1 uses trajectory vectors representing intensity features in all time directions and pre-learned recognition parameters to determine whether the sound of a device is normal or abnormal.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2015 / 068446 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] When generating a machine learning model that uses sound data to determine anomalies, it is considered possible to quantify the sound data using physical quantities such as sound pressure (dB) representing the magnitude of the sound data and frequency (Hz) representing the height of the sound data to generate the machine learning model. While generating a machine learning model can be efficient by visualizing the sound data, the invention described in Patent Document 1 does not mention visualizing sound data.

[0008] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing device and an image processing method for converting audio data into images.

[0009] Technical solutions to solve problems

[0010] An image processing device according to one embodiment of the present invention calculates a fundamental frequency component and a higher harmonic component corresponding to the fundamental frequency component contained in sound data, converts the fundamental frequency component and the higher harmonic component into image data, and generates a sound image in which the fundamental frequency component and the higher harmonic component converted into image data are adjacently arranged.

[0011] Effects of the Invention

[0012] According to the present invention, by visualizing audio data, a machine learning model of the audio data can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic configuration diagram of an image processing device 1 according to the first embodiment of the present invention.

[0014] Figure 2A This is an example of an image of audio data according to the first embodiment of the present invention.

[0015] Figure 2B This is an example of frequency analysis results.

[0016] Figure 3 This is an example of a machine learning model displayed on the display 50.

[0017] Figure 4 This is a flowchart for explaining an operation example of the image processing device 1 according to the first embodiment of the present invention.

[0018] Figure 5 It is a schematic configuration diagram of an image processing device 1 according to a second embodiment of the present invention.

[0019] Figure 6 This is an example of an image of audio data according to the second embodiment of the present invention.

[0020] Figure 7 This is a flowchart for explaining an operation example of the image processing device 1 according to the second embodiment of the present invention.

[0021] Figure 8 It is a schematic configuration diagram of an image processing device 1 according to a third embodiment of the present invention.

[0022] Figure 9 This is an example of an image of audio data according to the third embodiment of the present invention.

[0023] Figure 10 This is a flowchart for explaining an operation example of the image processing device 1 according to the third embodiment of the present invention.

[0024] Figure 11 It is a schematic configuration diagram of an image processing device 1 according to a fourth embodiment of the present invention.

[0025] Figure 12 This is an example of an image of audio data according to the fourth embodiment of the present invention.

[0026] Figure 13 This is a flowchart for explaining an operation example of the image processing device 1 according to the fourth embodiment of the present invention.

[0027] Figure 14 It is a schematic configuration diagram of an image processing device 1 according to a fifth embodiment of the present invention.

[0028] Figure 15 This is a diagram showing an example of a method for cutting out an image of audio data.

[0029] Figure 16 This is a flowchart for explaining an operation example of the image processing device 1 according to the fifth embodiment of the present invention.

[0030] Figure 17 It is a schematic configuration diagram of an image processing device 1 according to a sixth embodiment of the present invention.

[0031] Figure 18A This is a flowchart for explaining an operation example of the image processing device 1 according to the sixth embodiment of the present invention.

[0032] Figure 18B This is a flowchart for explaining an operation example of the image processing device 1 according to the sixth embodiment of the present invention. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same parts are denoted by the same reference numerals and their description will be omitted.

[0034] (First embodiment)

[0035] (Configuration Example of Image Processing Device)

[0036] Reference Figure 1 , a configuration example of the image processing device 1 according to the first embodiment will be described. Figure 1 As shown, the image processing apparatus 1 includes a controller 20 , a storage device 40 , and a display 50 .

[0037] The controller 20 acquires sound data via the microphone 10 and analyzes the acquired sound data. The controller 20 is a general-purpose microcomputer having a CPU (central processing unit), a memory, and an input / output unit. A computer program for functioning as the image processing device 1 is installed on the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits possessed by the image processing device 1. In addition, here, an example of realizing the plurality of information processing circuits possessed by the image processing device 1 by software is shown, but of course, dedicated hardware for performing each information processing shown below can also be prepared to constitute an information processing circuit. In addition, a plurality of information processing circuits can also be constituted by separate hardware. The controller 20 has a sound data acquisition unit 21, a frequency characteristic analysis unit 22, a basic frequency calculation unit 23, a sound data image generation unit 24, and a machine learning model generation unit 25 as a plurality of information processing circuits.

[0038] As described above, the sound data acquisition unit 21 acquires sound data via the microphone 10. The sound data acquired by the sound data acquisition unit 21 is converted into an electrical signal and processed as time-series data. Furthermore, the sound data acquired by the sound data acquisition unit 21 is labeled as normal or abnormal and used as training data for machine learning. For example, the sound data is sound data from machinery used in a factory.

[0039] The frequency characteristics analysis unit 22 analyzes the frequency characteristics of the sound data acquired by the sound data acquisition unit 21. Signal processing, such as FFT (Fast Fourier Transform), is used as an analysis method. The FFT converts time-series data into frequency data, obtaining a "frequency-energy characteristic."

[0040] The fundamental frequency calculation unit 23 calculates the fundamental frequency of the audio data using the "frequency-energy characteristics" obtained by the frequency characteristics analysis unit 22. The fundamental frequency calculation unit 23 also calculates harmonics corresponding to the calculated fundamental frequency.

[0041] The audio data image generating unit 24 generates an image of audio data using the fundamental frequency calculated by the fundamental frequency calculating unit 23 .

[0042] The machine learning model generation unit 25 uses the image generated by the sound data image generation unit 24 to generate an optimal machine learning model for determining whether the sound data is normal or abnormal. The machine learning model generation unit 25 verifies the performance of multiple machine learning algorithms for determining whether the sound data is normal or abnormal through cross-validation and outputs the model with the best performance. The machine learning model generation unit 25 may also output the model along with other models.

[0043] The storage device 40 stores the “frequency-energy characteristics” obtained by the frequency characteristics analysis unit 22 , the sound data image generated by the sound data image generation unit 24 , the machine learning model generated by the machine learning model generation unit 25 , and the like.

[0044] The machine learning model generated by the machine learning model generation unit 25 and its prediction performance are displayed on the display 50.

[0045] Next, refer to Figure 2A 、 Figure 2B , an example of an image generated by the audio data image generating unit 24 will be described.

[0046] Figure 2A Figure 60 is an example of an image generated by the sound data image generation unit 24. First, the method for generating image 60 will be described. Details of image 60 will be described later. The sound data acquisition unit 21 acquires sound data from any machine and converts the sound data into an electrical signal. Image 60 is an image obtained by converting sound data into image data and can also be referred to as a sound image.

[0047] The frequency characteristic analysis unit 22 analyzes the electrical signal using FFT. Figure 2B An example of the analysis result is shown in FIG. Figure 2BThe vertical axis represents the sound pressure (dB(A)), and the horizontal axis represents the frequency (Hz). Through FFT, we can get Figure 2B The "frequency-energy characteristics" shown.

[0048] The fundamental frequency calculation unit 23 uses Figure 2B The "frequency-energy characteristics" shown in the figure calculate the fundamental frequency of the sound data. The fundamental frequency means the frequency of the lowest frequency component when the signal (here, the sound data converted into an electrical signal) is represented by the synthesis of sine waves. The fundamental frequency is sometimes called the fundamental wave. According to FFT, Figure 2B As shown in FIG, there are generally multiple peak frequencies (peaks of the spectrum). The calculation method of the fundamental frequency is not particularly limited, but for example Figure 2B As shown, the minimum peak frequency among multiple peak frequencies can also be calculated ( Figure 2B 400Hz in) as the basic frequency.

[0049] As another calculation method, the frequency between peaks can also be calculated as the fundamental frequency. Figure 2B As shown, large peaks appear at 1200 Hz and 1600 Hz. The frequency between these peaks (1600-1200=400 Hz) can also be calculated as the fundamental frequency.

[0050] Furthermore, as another calculation method, the difference in frequency between the peaks may be obtained, and when the difference is the same as the minimum peak frequency, the frequency may be calculated as the basic frequency. Figure 2B As shown, since the difference in frequency between the peaks (400 Hz mentioned above) is the same as the minimum peak frequency (400 Hz), the fundamental frequency calculation unit 23 can also calculate this 400 Hz as the fundamental frequency. Here, "the difference is the same as the minimum peak frequency" does not necessarily mean that they are completely the same. "The difference is the same as the minimum peak frequency" means that they are substantially the same (considered to be approximately the same). In the first embodiment, the fundamental frequency is assumed to be 400 Hz for explanation. In addition, for one piece of sound data, there is one fundamental frequency.

[0051] Next, the fundamental frequency calculation unit 23 calculates the higher harmonics corresponding to the calculated fundamental frequency. Higher harmonics are frequency components that are integer multiples of a certain frequency component (here, the fundamental frequency component) with respect to the fluctuation. Higher harmonics are sometimes called overtones in the field of music. Figure 2B As shown, large peaks are observed at frequencies that are integer multiples (2 times, 3 times, etc.) of the fundamental frequency (400 Hz). The fundamental frequency calculation unit 23 calculates frequencies that are integer multiples of the fundamental frequency (800 Hz, 1200 Hz, etc.) as harmonics corresponding to the fundamental frequency.

[0052] The sound data image generating unit 24 uses the fundamental frequency and higher harmonics calculated by the fundamental frequency calculating unit 23, as shown in FIG. Figure 2A As shown, a two-dimensional image such as a so-called heat map is generated. Figure 2A The vertical axis represents overtones, and the horizontal axis represents frequencies. More specifically, the horizontal axis represents frequencies from 0 to 400 Hz. The overtones on the vertical axis mean overtones of the fundamental frequency (400 Hz). Image 60 is formed by a plurality of pixels representing corresponding frequencies. In image 60, below pixel 60a representing 400 Hz, pixel 60b representing 800 Hz (twice 400 Hz) is adjacently arranged. Similarly, below pixel 60b representing 800 Hz, pixel 1200 Hz (three times 400 Hz) is adjacently arranged. Moreover, a pixel representing 8400 Hz (21 times 400 Hz) is arranged at the lower right of image 60. Pixels are arranged in order of frequency on the horizontal and vertical axes. Image 60 can also be said to be image data consisting of a two-dimensional matrix in which a two-dimensional prescribed area (an area in which pixels 60a and 60b are arranged) is set for each frequency component and these prescribed areas are adjacently arranged vertically and horizontally.

[0053] In this way, the sound data image generating unit 24 converts the fundamental frequency component and the higher harmonic component calculated by the fundamental frequency calculating unit 23 into image data. The fundamental frequency component converted into image data is represented as pixel 60a (400 Hz). The higher harmonic component converted into image data is represented as pixel 60b (800 Hz). Figure 2A As shown, an image 60 (sound image) is generated by arranging the fundamental frequency component (pixel 60a of 400 Hz) and the higher harmonic component (pixel 60b of 800 Hz) converted into image data adjacent to each other. The sound image is obtained by converting the fundamental frequency component and the higher harmonic component contained in the sound data into an image through a predetermined process.

[0054] The shape of the image 60 is shown as a rectangle, but it is not limited to this. For example, the image 60 can also be comb-shaped to improve the resolution. In addition, each pixel is also displayed as a rectangle, but it is not limited to this. In addition, each pixel is separated from each other and arranged discretely, but it is not limited to this. Each pixel can also be arranged continuously without being separated from each other. The vertical axis and the horizontal axis can also be swapped. Figure 2A In FIG, the fundamental frequency component and the higher harmonic components corresponding to the fundamental frequency component are arranged adjacent to each other on the vertical axis, but they may be arranged adjacent to each other on the horizontal axis.

[0055] exist Figure 2AIn the example, each pixel is set to a brightness or color corresponding to the amplitude of the sound data. For example, pixels 60a and 60b with high sound pressure are set to a dark color (e.g., red) as in a heat map. Other pixels (e.g., pixel 60c) are set to a light color (e.g., green) as in a heat map. In addition, pixels 60d and 60e are an example of a display indicating an abnormality of the sound data, and the brightness or color is set differently from other pixels. By setting the brightness or color of the pixel, the operator can understand the intensity, normality, abnormality, etc. of the sound data at a glance.

[0056] Next, refer to Figure 3 , the machine learning model will be explained. The machine learning model generation unit 25 uses the image 60 generated by the sound data image generation unit 24 to generate an optimal machine learning model for determining whether the sound data is normal or abnormal. As described above, labels indicating normality or abnormality are assigned to the sound data. Therefore, the machine learning in the first embodiment is so-called "supervised learning."

[0057] The machine learning model generation unit 25 generates a machine learning model using a well-known machine learning algorithm. Examples of machine learning algorithms include decision trees, random forests, GBTs (Gradient Boosted Trees), GLRs (General Linear Regression), SVMs (Support Vector Machines), and deep learning. However, the present invention is not limited to these algorithms, and any algorithm that can perform supervised learning may be used.

[0058] The prediction performance of machine learning algorithms is compared using any combination of one or more metrics, such as accuracy, precision, and recall, that indicate the performance of the machine learning algorithm. Based on the comparison results, the model with the highest prediction performance is selected.

[0059] like Figure 3 As shown, the machine learning model generation unit 25 displays the generated machine learning model and its prediction performance on the display 50. The generated machine learning model (decision tree, etc.) and its prediction performance (70%, etc.) are displayed on the left side of the display 50. In addition, the details of the model with the highest prediction performance (deep learning) are displayed on the right side of the display 50. As details of deep learning, the form, number of learning times, ratio of normal to abnormal data, number of intermediate layers, and confusion matrix (Confusion Matrix) obtained by cross-validation are displayed.

[0060] Next, refer to Figure 4 An operation example of the image processing device 1 according to the first embodiment will be described with reference to the flowchart of FIG.

[0061] In step S101, the sound data acquisition unit 21 acquires sound data via the microphone 10. The acquired sound data is converted into an electrical signal and processed as time series data. The process proceeds to step S103, and the frequency characteristic analysis unit 22 uses FFT to analyze the electrical signal acquired in step S101. The "frequency-energy characteristic" (refer to Figure 2B ).

[0062] The process proceeds to step S105, and the fundamental frequency calculation unit 23 calculates the fundamental frequency of the sound data using the "frequency-energy characteristics" obtained in step S103. The method for calculating the fundamental frequency is the same as the method described above. The process proceeds to step S107, and the sound data image generation unit 24 generates an image 60 (see FIG. 6 ) such as a heat map using the fundamental frequency calculated in step S105. Figure 2A ). In the image 60, the fundamental frequency component and the higher harmonic components corresponding to the fundamental frequency component are arranged adjacent to each other.

[0063] The process proceeds to step S109, where the image 60 generated in step S107 is displayed on the display 50. Each pixel forming image 60 is set to a brightness or color corresponding to the amplitude of the sound data. This allows an operator viewing image 60 to understand at a glance the intensity of the sound data, and whether it is normal or abnormal.

[0064] The process proceeds to step S111, where the machine learning model generation unit 25 selects an algorithm for generating a machine learning model. Algorithms available for selection include decision tree, random forest, GBT (Gradient Boosted Tree), GLR (General Linear Regression), SVM (Support Vector Machine), and deep learning.

[0065] The process proceeds to step S113, and the machine learning model generation unit 25 generates a machine learning model using the algorithm selected in step S111. The process proceeds to step S115, and the machine learning model generation unit 25 displays the generated machine learning model and its prediction performance on the display 50 (see Figure 3 ). A series of processes are repeatedly executed until completion (step S117).

[0066] (Effect)

[0067] As described above, according to the image processing device 1 of the first embodiment, the following effects are achieved.

[0068] The fundamental frequency calculation unit 23 calculates the fundamental frequency component and the higher harmonic components corresponding to the fundamental frequency component contained in the sound data. The sound data image generation unit 24 converts the fundamental frequency component and the higher harmonic components calculated by the fundamental frequency calculation unit 23 into image data. The sound data image generation unit 24 generates the fundamental frequency component ( Figure 2A Pixel 60a) and higher harmonic components ( Figure 2A The image 60 (sound image) is formed by arranging pixels 60b adjacent to each other. Generating such an arranged sound image enables efficient generation of a machine learning model. This allows for clear display or processing of time-series sound signals or sensory quantities such as timbre that cannot be quantified using FFT-based spectral characteristics.

[0069] Furthermore, the fundamental frequency component (pixel 60a) and the higher harmonic component (pixel 60b) have a harmonic relationship. For single-tone sounds like car horns and string instruments, the harmonics affect the timbre. According to the first embodiment, this relationship can be displayed as a sound image.

[0070] The audio data image generator 24 arranges the fundamental frequency component (pixel 60a) and the harmonic component (pixel 60b) adjacent to each other on the vertical or horizontal axis of the audio image, thereby displaying the relationship between the fundamental frequency and the harmonic component as a two-dimensional audio image.

[0071] The audio data image generator 24 converts the fundamental frequency component and the higher harmonic component into image data with brightness or color set according to the amplitude of the audio data, thereby generating an audio image. This allows the intensity of the audio to be easily understood and displayed as an audio image.

[0072] The audio data image generating unit 24 arranges the plurality of frequency components of the audio data in order of frequency on the vertical axis or horizontal axis of the audio image, thereby displaying the audio data as a two-dimensional audio image.

[0073] (Second embodiment)

[0074] Next, refer to Figure 5 The second embodiment of the present invention will now be described. The second embodiment differs from the first embodiment in that the controller 20 includes a scale setting unit 26. Structures that overlap with those in the first embodiment are referenced by reference numerals and their descriptions are omitted. The following description will focus on the differences.

[0075] The scale setting unit 26 uses the "frequency-energy characteristics" obtained by the frequency characteristics analysis unit 22 to set the twelve scales of "C, C#, D, D#, E, F, G, G#, A, A#, B, B#, (C)" corresponding to "do, re, mi, fa, sol, la, si (do)" used in music, and octaves (1 to 10 octaves) as their overtones. The scale setting unit 26 uses the twelve scales and octaves to classify sounds of different frequencies into the twelve scales.

[0076] Next, refer to Figure 6 , an example of an image generated by the audio data image generating unit 24 will be described.

[0077] Figure 6 Reference numeral 61 is an example of an image generated by the sound data image generating unit 24. The sound data image generating unit 24 uses the twelve scales and octaves set by the scale setting unit 26 and the frequencies classified by the scale setting unit 26, and arranges pixels representing frequencies corresponding to the twelve scales to generate the image 61. Figure 6 The horizontal axis represents the twelve scales, and the vertical axis represents the octave (harmonics of the frequency). The first row of the image 61 represents the scale of 32Hz to 65Hz, and the harmonics become harmonics as they go downward. In addition, in the example shown in FIG2, the harmonics are explained as integer multiples of the fundamental frequency, but in Figure 6 In the example shown, the overtone is twice the previous frequency.

[0078] In the second embodiment, the fundamental frequency is not 400 Hz but an arbitrary value. The scale corresponding to the pixel 60a representing the fundamental frequency is "A". The scale corresponding to the pixel 60b representing the higher harmonic corresponding to the fundamental frequency is also "A". Figure 6 As shown, pixel 60a and pixel 60b are arranged adjacent to each other. By generating such an image 61, sound data can be represented as a two-dimensional image that matches the human auditory characteristics.

[0079] Image 61 also has the same meaning as image 60 (see Figure 2A ) is similarly displayed in a rectangular shape, but is not limited thereto. Items that can be changed in image 60 also apply to image 61.

[0080] The machine learning model generation unit 25 generates an optimal machine learning model for determining whether the audio data is normal or abnormal using the image 61. The details are the same as those of the first embodiment, and therefore description thereof will be omitted.

[0081] Next, refer to Figure 7 The following describes an example of the operation of the image processing device 1 according to the second embodiment. However, since the processing of steps S201 to S205 and S209 to S219 is the same as that of Figure 4 The processes shown in steps S101 to S105 and S107 to S117 are the same, so their description is omitted.

[0082] In step S207, the scale setting unit 26 sets twelve scales and octaves as their harmonics using the frequency-energy characteristics obtained in step S203. The scale setting unit 26 classifies sounds of various frequencies into twelve scales using the twelve scales and octaves.

[0083] (Effect)

[0084] In the second embodiment, the fundamental frequency component and the higher harmonic components have the same scale. According to the second embodiment, music-related phenomena such as scales and octaves can be displayed as images.

[0085] (Third embodiment)

[0086] Next, refer to Figure 8 The third embodiment of the present invention will now be described. The third embodiment differs from the first embodiment in that the controller 20 includes a critical band setting unit 27. Components that overlap with those in the first embodiment are referenced by reference numerals, and their descriptions are omitted. The following description focuses on the differences.

[0087] The critical band setting unit 27 uses the "frequency-energy characteristics" obtained by the frequency characteristics analysis unit 22 to set the twelve musical scales of "C, C#, D, D#, E, F, G, G#, A, A#, B, B#, (C)" corresponding to "do, re, mi, fa, sol, la, si (do)" used in music, and critical bands (bands numbered 1 to 24) that represent human auditory characteristics. Using the twelve musical scales and critical bands, the critical band setting unit 27 classifies sounds of each frequency into the twelve musical scales.

[0088] The critical band is defined as the maximum frequency band where the sound level of band noise remains constant regardless of bandwidth, while maintaining a constant band sound pressure level. Alternatively, the critical band is defined as the minimum frequency bandwidth of the band noise at which, when the bandwidth is increased while maintaining a constant band noise spectrum, a pure tone with a sound pressure level equal to the band noise's center frequency is just audible.

[0089] Next, refer to Figure 9 , an example of an image generated by the audio data image generating unit 24 will be described.

[0090] Figure 9The image 62 shown is an example of an image generated by the sound data image generating unit 24. The sound data image generating unit 24 uses the twelve scales and critical bands set by the critical band setting unit 27, and the frequencies classified by the critical band setting unit 27, and arranges pixels representing frequencies corresponding to the twelve scales to generate the image 62.

[0091] Figure 9 The horizontal axis represents the twelve scales, and the vertical axis represents the critical bands (band numbers are 1 to 24). Figure 9 In, omitted Figure 6 As shown in the figure, "C, C#, D, D#, E, F, G, G#, A, A#, B, B#, C". In addition, Figure 9 The horizontal axis represents 32Hz to 130Hz, a combination of one and two octaves. The critical band number indicates the minimum frequency of that critical band. If the vertical axis is represented by frequency, the frequency increases from the top in the order of 100Hz, 200Hz, 300Hz, 400Hz, and 510Hz. The reason why the frequency increases irregularly on the vertical axis is because it mimics the characteristics of human hearing. Figure 9 The dashed line shows the sensitivity characteristic relative to the sound volume. The band surrounded by the dashed line shows that the sensitivity characteristic is the same.

[0092] In the third embodiment, the fundamental frequency is not 400 Hz but an arbitrary value. Figure 9 As shown, pixels 60a representing the fundamental frequency and pixels 60b representing the corresponding harmonics are arranged adjacent to each other. By generating an image 62 in which the fundamental frequency component and the harmonic components, which have a relationship related to the critical band of human hearing, are arranged adjacent to each other, it is possible to represent sound data as a two-dimensional image that matches the characteristics of human hearing.

[0093] Image 62 also has the same Figure 2A ) is similarly displayed in a rectangular shape, but is not limited thereto. Items that can be changed in image 60 also apply to image 62.

[0094] The machine learning model generation unit 25 generates an optimal machine learning model for determining whether the audio data is normal or abnormal using the image 62. The details are the same as those of the first embodiment, and therefore description thereof will be omitted.

[0095] Next, refer to Figure 10 The following describes an example of the operation of the image processing apparatus 1 according to the third embodiment. However, since the processing of steps S301 to S305 and S309 to S319 is the same as that of the image processing apparatus 1 according to the third embodiment, Figure 4 The processes shown in steps S101 to S105 and S107 to S117 are the same, so their description is omitted.

[0096] In step S307, the critical band setting unit 27 sets twelve scales and critical bands using the frequency-energy characteristics obtained in step S303. The critical band setting unit 27 classifies sounds of various frequencies into twelve scales using the twelve scales and critical bands.

[0097] (Effect)

[0098] In the third embodiment, the fundamental frequency component and the higher harmonic components have a relationship related to the critical band of human hearing. According to the third embodiment, such a relationship can be displayed as an image.

[0099] (Fourth embodiment)

[0100] Next, refer to Figures 11-12 , the fourth embodiment of the present invention will be described. The fourth embodiment is different from the first embodiment in that the controller 20 includes a different time image generating unit 28 and a three-dimensional image generating unit 29 to replace the sound data image generating unit 24 (see Figure 11 ) The same components as those in the first embodiment are denoted by reference numerals and their descriptions are omitted. The following description will focus on the differences.

[0101] Figure 12 The image 60 shown is Figure 2A The different-time image generating unit 28 generates an image 60 using the fundamental frequency and harmonics calculated by the fundamental frequency calculating unit 23. Furthermore, the different-time image generating unit 28 performs the same processing at each predetermined time to generate images 63 to 65. The three-dimensional image generating unit 29 generates a three-dimensional image 70 using the images 60, 63 to 65 generated by the different-time image generating unit 28 at each predetermined time. Specifically, the three-dimensional image generating unit 29 adds a time axis to the images 60, 63 to 65 (two-dimensional images) to generate the three-dimensional image 70. The items that can be changed in the image 60 also apply to the three-dimensional image 70. Due to the calculation characteristics of the FFT, the data at each predetermined time is generally calculated with overlap (interleaving) with the data before and after it on the time axis, and the value at the predetermined time is calculated using data of a time length such as a predetermined frequency resolution.

[0102] The machine learning model generation unit 25 generates an optimal machine learning model for determining whether the audio data is normal or abnormal using the three-dimensional image 70. The details are the same as those of the first embodiment, and therefore description thereof will be omitted.

[0103] Next, refer to Figure 13 The following describes an example of the operation of the image processing apparatus 1 according to the fourth embodiment. However, since the processing of steps S401 to S405 and S413 to S419 is similar to that of Figure 4The processes shown in steps S101 to S105 and S111 to S117 are the same, so their description is omitted.

[0104] In step S407, the time-varying image generation unit 28 generates images 60, 63, and 65 at each predetermined time interval using the fundamental frequency and higher harmonics obtained in step S405. The process then proceeds to step S409, where the three-dimensional image generation unit 29 generates a three-dimensional image 70 using the images 60, 63, and 65 generated at each predetermined time interval in step S407. The process proceeds to step S411, where the three-dimensional image 70 generated in step S409 is displayed on the display 50. Each pixel forming the three-dimensional image 70 is set to a brightness or color corresponding to the amplitude of the audio data. This allows an operator viewing the three-dimensional image 70 to understand at a glance the intensity of the audio data and whether it is normal or abnormal.

[0105] (Effect)

[0106] According to the fourth embodiment, audio data can be displayed as a three-dimensional image.

[0107] (Fifth embodiment)

[0108] Next, refer to Figures 14-15 , the fifth embodiment of the present invention will be described. The fifth embodiment is different from the fourth embodiment in that the controller 20 further includes a frequency setting unit 30 and an image cutting unit 31 (see Figure 14 ) The same components as those in the fourth embodiment are denoted by reference numerals and their descriptions are omitted. The following description will focus on the differences.

[0109] The frequency setting unit 30 sets a frequency extracted from the three-dimensional image 70. The frequency setting unit 30 may set an arbitrary frequency, and may set a basic frequency.

[0110] The image cutting unit 31 cuts out pixels related to the frequency set by the frequency setting unit 30. Specifically, Figure 15 As shown, the image cutting unit 31 cuts out data 80 of a plurality of pixels corresponding to the frequency set by the frequency setting unit 30. The image cutting unit 31 generates a spectrogram using the cut-out data 80. A spectrogram is a graph in which frequency spectra of frequency characteristics are arranged on a time axis and intensity is represented by color, like a heat map.

[0111] The machine learning model generation unit 25 generates an optimal machine learning model for determining whether the audio data is normal or abnormal using the spectrogram. Detailed descriptions are the same as those in the first embodiment, so their explanations are omitted.

[0112] Next, refer to Figure 16The flowchart of the fifth embodiment will be used to describe an example of the operation of the image processing device 1. However, since the processing of steps S501 to S509 and S515 to S521 is the same as that of the image processing device 1 of the fifth embodiment, Figure 13 The processes shown in steps S401 to S409 and S413 to S419 are the same, so their description is omitted.

[0113] In step S511, the frequency setting unit 30 sets the frequency to be extracted from the three-dimensional image 70. The process proceeds to step S513, and the image cutting unit 31 cuts out the pixels related to the frequency set in step S511. The image cutting unit 31 generates a spectrogram using the cut-out pixels.

[0114] (Effect)

[0115] According to the fifth embodiment, analysis using a spectrogram can be performed.

[0116] (Sixth embodiment)

[0117] Next, refer to Figure 17 The sixth embodiment of the present invention will now be described. The sixth embodiment differs from the first embodiment in that the controller 20 includes a new sound data acquisition unit 32, an image processing unit 33, and a determination unit 34. Components that overlap with those in the first embodiment are referenced by reference numerals, and their descriptions are omitted. The following description focuses on the differences.

[0118] The new audio data acquisition unit 32 acquires new audio data via a microphone 11 different from the microphone 10. The microphones 10 and 11 are respectively mounted on the same type of machine. The audio data image generation unit 24 generates an image of the new audio data.

[0119] The image processing unit 33 uses the image of the new audio data as input data for the machine learning model generated by the machine learning model generation unit 25. The image processing unit 33 outputs an index such as the degree of image consistency using a predetermined image processing method.

[0120] Determination unit 34 compares the output value from image processing unit 33 with a preset threshold value. If the output value exceeds the threshold value, determination unit 34 determines that the audio data is normal. On the other hand, if the output value is below the threshold value, determination unit 34 determines that the audio data is abnormal. The determination method is not limited to the method using a threshold value; other methods may also be used.

[0121] The determination result of the determination unit 34 is displayed on the display 50. Although not shown in the figure, the determination result of the determination unit 34 may be notified by voice via a speaker. Alternatively, when the determination result of the determination unit 34 is abnormal, a red turn signal may be turned on.

[0122] Next, refer to Figures 18A-18B The flowchart of the sixth embodiment is used to describe an operation example of the image processing device 1. However, since the processing of steps S601 to S613 is the same as that of Figure 4 The processes shown in steps S101 to S113 are the same, so their description is omitted.

[0123] In step S615, the new sound data acquisition unit 32 acquires new sound data via the microphone 11. The acquired new sound data is converted into an electrical signal and processed as time-series data. The process proceeds to step S617, where the frequency characteristics analysis unit 22 uses FFT to analyze the electrical signal acquired in step S615. The FFT obtains a "frequency-energy characteristic."

[0124] The process proceeds to step S619, where the fundamental frequency calculation unit 23 calculates the fundamental frequency of the new sound data using the "frequency-energy characteristics" obtained in step S617. The process proceeds to step S621, where a determination is made as to whether the fundamental frequency calculated in step S619 matches the fundamental frequency of the machine learning model generated in step S613. This determination is made because, if the fundamental frequencies differ, the machine learning model cannot perform normal / abnormal determination processing. Furthermore, "consistent fundamental frequencies" means substantial consistency.

[0125] If the fundamental frequencies do not match ("No" in step S621), the message "Because the fundamental frequencies do not match, determination processing cannot be performed" is displayed on the display 50, and the process proceeds to step S631. On the other hand, if the fundamental frequencies match ("Yes" in step S621), the process proceeds to step S623, and the sound data image generator 24 generates a new sound data image using the fundamental frequencies calculated in step S619.

[0126] Processing proceeds to step S625, where the image processing unit 33 uses the new audio data image generated in step S623 as input data for the machine learning model. Using a prescribed image processing method, the image processing unit 33 outputs metrics such as image consistency. The determination unit 34 compares the output value from the image processing unit 33 with a preset threshold value to determine whether the audio data is normal or abnormal.

[0127] The process proceeds to step S627, where the determination result of step S625 is displayed on the display 50. The process proceeds to step S629, where the file name of the new sound data, the name of the machine learning model, the processing execution time, the value of the fundamental frequency, the determination result, etc. are stored in the storage device 40. This series of processes is repeated until completion (step S631). Furthermore, a message such as "Normal / Abnormal Determination Process Completed" may be displayed on the display 50 at the end of the process.

[0128] (Effect)

[0129] According to the sixth embodiment, it is possible to determine whether other audio data is normal or abnormal using a machine learning model obtained by imaging audio data.

[0130] The functions described in the above embodiments may be implemented by one or more processing circuits. Processing circuits include programmed processing devices such as processing devices including circuits. Processing circuits also include devices such as application-specific integrated circuits (ASICs) or circuit components configured to perform the described functions.

[0131] While the embodiments of the present invention have been described above, the description and drawings forming part of this disclosure should not be construed as limiting the present invention. Various alternative embodiments, examples, and operational techniques will be apparent to those skilled in the art based on this disclosure.

[0132] In the above embodiment, a machine learning model is used as a method for determining the normality or abnormality of other sound data, but the present invention is not limited to this. For example, it is also possible to determine abnormalities in images (sound data) by comparing fundamental frequency components and higher harmonic components with other frequency components. This makes it possible to determine the normality or abnormality of sound data in cases where there is no overtone relationship, such as in the critical band.

[0133] Alternatively, the determination unit 34 may determine abnormality in a predetermined sound included in the sound data using the image 60 (sound image).

[0134] Image 60 (sound image) may also include fundamental frequency components, higher harmonic components, and other frequency components converted into image data, and may be composed of a two-dimensional matrix with a predetermined region set for each frequency component. Furthermore, the other frequency components refer to frequency components other than the fundamental frequency components and higher harmonic components.

[0135] Description of labels

[0136] 1 Image processing device

[0137] 20 Controller

[0138] 21 Sound Data Acquisition Unit

[0139] 22 Frequency Characteristics Analysis Section

[0140] 23 Basic frequency calculation unit

[0141] 24 Sound data image generation unit

[0142] 25 Machine Learning Model Generation Department

[0143] 26 Scale Setting Section

[0144] 27 Critical Band Setting Section

[0145] 28. Image generation unit at different times

[0146] 30 Frequency setting unit

[0147] 31 Image cutout section

[0148] 32 New Sound Data Acquisition Department

[0149] 33 Image Processing Unit

[0150] 34 Judgment Department

[0151] 40 Storage Devices

[0152] 50 Display

Claims

1. An image processing device that acquires sound data and analyzes the acquired sound data, characterized in that: The image processing device comprises: a sound data acquisition unit that acquires sound data from a microphone; and controller, The controller includes: a frequency characteristic analyzing unit for analyzing the frequency characteristics of the sound data; a fundamental frequency calculation unit that calculates a fundamental frequency component included in the sound data and a higher harmonic component corresponding to the calculated fundamental frequency component; A sound data image generating unit converts the calculated fundamental frequency component, the calculated higher harmonic component, and other frequency components present in the sound data and different from the calculated fundamental frequency component and the higher harmonic component into image data, and further generates a sound image of a two-dimensional matrix including a predetermined area, wherein: the calculated fundamental frequency component and the calculated higher harmonic component are represented by respective pixels arranged adjacent to each other in respective first predetermined areas in frequency order along an axis of the two-dimensional matrix representing higher harmonic components, According to the amplitude of the sound data, the brightness or color of each pixel corresponding to the calculated fundamental frequency component and the calculated higher harmonic component is set. The second predetermined areas are adjacent to each other along the axis representing the frequency components of the two-dimensional matrix and are aligned with each of the first predetermined areas associated with the calculated fundamental frequency component or the calculated higher harmonic components, the second predetermined areas being associated with frequencies respectively included between 0 and the calculated fundamental frequency component and between the calculated fundamental frequency component and the calculated higher harmonic components in frequency order, The other frequency components converted into image data are represented by respective pixels arranged in corresponding second predetermined areas, and setting the brightness or color of each pixel corresponding to the other frequency components according to the respective amplitudes of the sound data of the other frequency components; and The machine learning model generation unit uses the generated sound images to generate a machine learning model to determine anomalies in different sound data.

2. The image processing device according to claim 1, wherein The fundamental frequency component and the higher harmonic component have an overtone relationship, have the same musical scale, or have a relationship related to a critical band of human hearing.

3. The image processing device according to claim 1 or 2, characterized in that The controller determines abnormalities in the different acquired sound data using a sound image generated from the different acquired sound data and a generated machine learning model.

4. An image processing method, characterized in that: Get sound data from the microphone; analyzing the frequency characteristics of the sound data by a frequency characteristics analyzing unit; calculating, by a fundamental frequency calculation unit, a fundamental frequency component included in the sound data and a higher harmonic component corresponding to the calculated fundamental frequency component; The sound data image generating unit converts the calculated fundamental frequency component, the calculated higher harmonic component, and other frequency components existing in the sound data and other than the calculated fundamental frequency component and the calculated higher harmonic component into image data. The sound data image generating unit generates a sound image including a two-dimensional matrix of predetermined areas, wherein: the calculated fundamental frequency component and the calculated higher harmonic components are represented by respective pixels arranged adjacent to each other in frequency order along an axis of the two-dimensional matrix representing higher harmonic components, According to the amplitude of the sound data, the brightness or color of each pixel corresponding to the calculated fundamental frequency component and the calculated higher harmonic component is set. The second predetermined areas are adjacent to each other along the axis representing the frequency components of the two-dimensional matrix and are aligned with each of the first predetermined areas associated with the calculated fundamental frequency component or the calculated higher harmonic components, the second predetermined areas being associated with frequencies respectively included between 0 and the calculated fundamental frequency component and between the calculated fundamental frequency component and the calculated higher harmonic components in frequency order, The other frequency components converted into image data are represented by respective pixels arranged in corresponding second predetermined areas, and setting the brightness or color of each pixel corresponding to the other frequency components according to the respective amplitudes of the sound data of the other frequency components; and The machine learning model generation unit generates a machine learning model using the generated sound images to determine abnormalities in different sound data.

Images