Information processing method, information processing device, and program
The information processing method and device streamline the extraction of target sounds from non-stationary sounds by allowing user-defined target range input and automatic similarity extraction, addressing the challenge of analysis accuracy and user effort in existing sound evaluation systems.
Patent Information
- Application Number
- PCT/JP2024/016760
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2024-05-01
- Publication Date
- 2024-11-21
AI Technical Summary
Existing sound evaluation devices face challenges in accurately analyzing target objects due to the presence of non-stationary sounds, which are difficult to extract and can decrease analysis accuracy, requiring significant effort to isolate the target sound.
An information processing method and device that acquires sound data from a target object, identifies non-stationary sounds, and allows users to input a target range for extraction, automatically extracting similar waveforms from the sound data, reducing user effort and improving extraction efficiency.
Enables the efficient extraction of target sounds from non-stationary sounds by automatically identifying similar waveforms, thereby reducing user input and enhancing the accuracy of sound analysis.
Smart Images

Figure JP2024016760_21112024_PF_FP_ABST
Abstract
Description
Information processing method, information processing device, and program
[0001] The present disclosure relates to an information processing method, an information processing device, and a program.
[0002] Patent Document 1 discloses a sound evaluation device that can separate and evaluate stationary sounds and non-stationary sounds from a plurality of sound sources generated from the device.
[0003] Japanese Patent Application Laid-Open No. 2009-236645
[0004] Non-stationary sounds may include noise, ambient environmental sounds, and the like, in addition to sounds from objects such as equipment. When analyzing an object using such non-stationary sounds, the accuracy of the analysis may be reduced. Therefore, it is desirable to extract the target sound from the non-stationary sounds, but extracting the target sound requires a great deal of effort.
[0005] Therefore, the present disclosure provides an information processing method, an information processing device, and a program that can assist in extracting a target sound from a non-stationary sound.
[0006] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, which acquires sound data obtained by collecting sound from an object, and if the sound data includes a non-stationary sound, presents a waveform of the non-stationary sound, acquires input of a first target range that is a range to be extracted from the presented waveform, and extracts one or more first similar ranges from the sound data that include waveforms similar to the acquired waveform of the first target range.
[0007] An information processing device according to one aspect of the present disclosure includes a first acquisition unit that acquires sound data obtained by collecting sound from an object, a presentation control unit that, if the sound data includes a non-steady sound, causes a waveform of the non-steady sound to be presented, a second acquisition unit that acquires an input of a first target range, which is a range to be extracted from the presented waveform, and an extraction unit that extracts one or more first similar ranges from the sound data that include waveforms similar to the acquired waveforms of the first target range.
[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described information processing method.
[0009] According to one aspect of the present disclosure, it is possible to realize an information processing method or the like that can assist in extracting a target sound from a non-stationary sound.
[0010] FIG. 1 is a block diagram showing a functional configuration of an information processing system according to an embodiment. FIG. 2 is a diagram showing an example of the waveform of a steady sound. FIG. 3 is a diagram showing an example of the waveform of a non-steady sound. FIG. 4 is a flowchart showing a sound registration operation in an information processing system according to an embodiment. FIG. 5A is a diagram showing collected sound data. FIG. 5B is a diagram showing a first target range input for the collected sound data. FIG. 5C is a diagram showing one or more first similarity ranges automatically extracted for a waveform in the first target range. FIG. 5D is a diagram showing a second target range input for the collected sound data. FIG. 5E is a diagram showing one or more second similarity ranges automatically extracted for a waveform in the second target range. FIG. 6 is a flowchart showing a learning operation in an information processing system according to an embodiment. FIG. 7 is a diagram for explaining a method for evaluating a generated machine learning model according to an embodiment. FIG. 8 is a flowchart showing a sound registration operation in an information processing system according to a first modification of the embodiment. FIG. 9 is a flowchart showing a sound registration operation in an information processing system according to a second modification of the embodiment. FIG. 10 is a flowchart showing a sound registration operation in an information processing system according to a third modification of the embodiment. 11A is a diagram showing a range of registration candidates automatically extracted from collected sound data, and FIG. 11B is a diagram showing the range of registration candidates after a part of the range of registration candidates has been deleted by a user.
[0011] An information processing method according to a first aspect of the present disclosure is an information processing method executed by a computer, which acquires sound data obtained by collecting sound from an object, and if the sound data includes a non-stationary sound, presents a waveform of the non-stationary sound, acquires input of a first target range, which is a range to be extracted from the presented waveform, and extracts one or more first similar ranges from the sound data that include waveforms similar to the acquired waveform of the first target range.
[0012] As a result, simply by the user inputting a target sound (a desired sound) into the sound data, sounds similar to the target sound can be automatically extracted from the acquired sound data including non-stationary sounds. In other words, since the user does not need to input a sound similar to the target sound, the user's effort to extract sounds from non-stationary sounds can be reduced. Therefore, an information processing method can be realized that can assist in extracting a target sound from non-stationary sounds.
[0013] Furthermore, for example, the information processing method according to the second aspect may be the information processing method according to the first aspect, in which, after extracting the one or more first similar ranges, an input of a second target range, which is another range to be extracted in the presented waveform, is acquired, and one or more second similar ranges including waveforms similar to the acquired waveforms of the second target range are extracted from the sound data.
[0014] This allows the user to simply input additional target sounds (desired sounds) into the sound data, and automatically extract sounds similar to the added target sounds from the acquired sound data containing non-stationary sounds.
[0015] Furthermore, for example, an information processing method according to a third aspect may be the information processing method according to the first or second aspect, wherein, when the variation of the one or more first similarity ranges based on a first similarity between the waveform in the first target range and each of the waveforms in the one or more first similarity ranges is smaller than a first threshold, a second target range having a similarity to the waveform in the first target range different from the first similarity is identified, and one or more second similarity ranges including waveforms similar to the identified waveforms in the second target range are extracted from the sound data.
[0016] As a result, when the variation in the extracted first similarity range is small, a second similarity range that has a large variation is extracted, thereby enabling automatic extraction of various target sounds, thereby realizing an information processing method that can further support extraction of target sounds from non-stationary sounds.
[0017] Furthermore, for example, an information processing method according to a fourth aspect is an information processing method according to any one of the first to third aspects, wherein the one or more first similarity ranges are ranges that include waveforms having a similarity to the waveforms in the first target range that is equal to or greater than a third threshold, and when the variation of the one or more first similarity ranges based on the first similarity between the waveforms in the first target range and each of the waveforms in the one or more first similarity ranges is greater than a second threshold, the third threshold may be changed to a fourth threshold that is greater than the third threshold, and the one or more first similarity ranges may be re-extracted based on the changed fourth threshold.
[0018] If the variation is large, there is a possibility that sounds other than the target object are included in the extracted first similar range. If the variation is large, the threshold value of the similarity is changed (for example, changed to a higher value), thereby making it possible to automatically exclude the first similar range that includes sounds other than the target object. Thus, it is possible to realize an information processing method that can further support the extraction of the target sound from non-stationary sounds.
[0019] Furthermore, for example, an information processing method according to a fifth aspect may be an information processing method according to any of the first to fourth aspects, and if the number of the one or more extracted first similar ranges is smaller than a fifth threshold, a second target range different from the first target range and the one or more first similar ranges may be identified, and one or more second similar ranges including waveforms similar to waveforms of the identified second target ranges may be extracted from the sound data.
[0020] This allows for automatic extraction of one or more additional second similarity ranges when the number of extracted first similarity ranges is small, thereby realizing an information processing method that can further assist in extracting a target sound from non-stationary sounds.
[0021] Furthermore, for example, an information processing method according to a sixth aspect is an information processing method according to any one of the first to fifth aspects, wherein the one or more first similarity ranges are ranges that include waveforms that have a similarity to the waveforms in the first target range that is equal to or greater than a third threshold, and when the number of the one or more extracted first similarity ranges is greater than a sixth threshold, the third threshold may be changed to a fourth threshold that is greater than the third threshold, and the one or more first similarity ranges may be re-extracted based on the changed fourth threshold.
[0022] As a result, when the number of extracted first similar ranges is large, the similarity threshold is changed (for example, increased) to automatically reduce the number of extracted first similar ranges, thereby realizing an information processing method that can further support extraction of a target sound from a non-stationary sound.
[0023] Furthermore, for example, the information processing method according to the seventh aspect may be an information processing method according to any one of the second, third, and fifth aspects, in which the extracted one or more first similarity ranges are presented, and after presenting the one or more first similarity ranges, the input of the second target range is obtained.
[0024] This allows the user to determine whether to additionally set a second target range after checking the first similarity range. In other words, it is possible to support the user in determining whether to additionally set the second target range. Therefore, it is possible to realize an information processing method that can further support extraction of a target sound from a non-stationary sound.
[0025] Furthermore, for example, the information processing method according to the eighth aspect may be an information processing method according to any one of the second, third, fifth, and seventh aspects, and may present the extracted one or more second similarity ranges.
[0026] This allows the user to check the sound that combines the extracted waveforms in the second similarity range. The user can make various decisions based on the sound in the second similarity range. Therefore, an information processing method can be realized that can further support the extraction of a target sound from non-stationary sounds.
[0027] Also, for example, an information processing method according to a ninth aspect may be an information processing method according to any one of the first to eighth aspects, in which a waveform of the sound data is displayed and the sound data is played back, and after the sound data is displayed and played back, the input of the first target range is obtained.
[0028] This allows the user to initially determine whether the acquired non-stationary sound can be used for sound registration. Therefore, it is possible to prevent processing such as extraction processing from being performed on non-stationary sounds that cannot be used for sound registration. This leads to a reduction in the amount of processing performed by the information processing device that executes the information processing method.
[0029] Furthermore, for example, an information processing method according to a tenth aspect may be an information processing method according to any one of the first to ninth aspects, in which a waveform in the first target range and a waveform in one or more first similar ranges are combined, and sound data including the combined waveform is played back.
[0030] This allows the user to confirm a sound that combines waveforms from the first target range and one or more first similar ranges. The user can make various decisions based on the sounds in the first similar ranges. Therefore, an information processing method can be realized that can better support the extraction of a target sound from non-stationary sounds.
[0031] Furthermore, for example, the information processing method according to the eleventh aspect may be an information processing method according to any one of the second, third, fifth, seventh and eighth aspects, in which waveforms of the first target range, the second target range, the one or more first similar ranges and the one or more second similar ranges are combined, and sound data including the combined waveforms is played back.
[0032] This allows the user to hear a sound in which the waveforms of the first target range, the second target range, one or more first similar ranges, and one or more second similar ranges are combined, which leads to a reduction in the amount of processing by the information processing device that executes the information processing method, compared to when the waveforms of the first target range, the second target range, one or more first similar ranges, and one or more second similar ranges are played back separately (without being combined).
[0033] Also, for example, an information processing method according to a twelfth aspect may be an information processing method according to any one of the first to eleventh aspects, in which the object is a manufacturing device, and the sound of the object may include sound picked up while the manufacturing device is operating.
[0034] This makes it possible to realize an information processing method that can assist in extracting target sounds from non-stationary sounds emitted by manufacturing equipment.
[0035] An information processing device according to a thirteenth aspect of the present disclosure includes a first acquisition unit that acquires sound data obtained by collecting sound from an object, a presentation control unit that, if the sound data includes a non-stationary sound, causes a waveform of the non-stationary sound to be presented, a second acquisition unit that acquires an input of a first target range that is a range to be extracted from the presented waveform, and an extraction unit that extracts one or more first similar ranges from the sound data that include waveforms similar to the acquired waveforms in the first target range. Also, a program according to a fourteenth aspect of the present disclosure is a program for causing a computer to execute the information processing method according to any one of the first to twelfth aspects.
[0036] This provides the same effect as the above-described information processing method.
[0037] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.
[0038] Hereinafter, the embodiments will be specifically described with reference to the drawings.
[0039] The embodiments and the like described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments and the like are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims are described as optional components.
[0040] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.
[0041] Furthermore, in this specification, terms indicating relationships between elements such as "same," as well as numerical values and numerical ranges, are not expressions that only express a strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about several percent (or about 10%).
[0042] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.
[0043] (Embodiment) Hereinafter, an information processing system according to the present embodiment will be described with reference to FIGS.
[0044] 1. Configuration of Information Processing System First, the configuration of an information processing system according to this embodiment will be described with reference to Figures 1 to 3. Figure 1 is a block diagram showing the functional configuration of an information processing system 1 according to this embodiment.
[0045] 1, the information processing system 1 is a support system for supporting the registration of sound data for training a machine learning model, and includes an information processing device 10, a sound collection device 20, and a display device 30. The information processing system 1 may further include a machine learning device 50. The information processing device 10 is communicably connected to each of the sound collection device 20, the display device 30, the sound output device 40, and the machine learning device 50.
[0046] The machine learning model receives sound data captured from an object as input and outputs information indicating the timing at which the object sound was emitted. The object is a device that emits sound when in operation, such as, but not limited to, a manufacturing device that processes an object. The following describes an example in which the object is a manufacturing device. Furthermore, registering sound data means storing the sound data as input data to be used when training the machine learning model.
[0047] The information processing device 10 includes an acquisition unit 11, a determination unit 12, a display control unit 13, a sound output control unit 14, a reception unit 15, an extraction unit 16, a processing unit 17, a storage unit 18, and an evaluation unit 19. The information processing device 10 has a CPU (Central Processing Unit) and a memory, and each function is realized by the CPU executing a program stored in the memory. The information processing device 10 may be realized, for example, by a computer or a server.
[0048] The acquisition unit 11 acquires sound data obtained by sound collection from a sound collection device 20 that collects sounds from the manufacturing equipment. The manufacturing equipment has multiple drive mechanisms, and sounds are generated when the multiple drive mechanisms operate or come into contact with each other. The acquisition unit 11 acquires sound data collected when such manufacturing equipment is operating. The sound data is waveform data (e.g., digital data) obtained by sampling the sounds from the manufacturing equipment. The acquisition unit 11 is configured to include, for example, a communication circuit (communication module). The acquisition unit 11 is an example of a first acquisition unit.
[0049] The sounds contained in the sound data acquired by the acquisition unit 11 can be roughly classified into steady sounds and non-steady sounds. Steady sounds and non-steady sounds will be described with reference to FIGS. 2 and 3.
[0050] FIG. 2 is a diagram showing an example of a waveform (sound waveform) of a steady sound. FIG. 2 illustrates a graph showing the sound volume of sound data W1 and W2 (sound data W1 and W2 in the upper row) and a graph showing the frequency spectrogram of sound data W1 and W2 (sound data W1 and W2 in the lower row) arranged vertically. The sound data W1 and W2 are sound data obtained by collecting steady sounds. Furthermore, the horizontal axis of the sound data W1 and W2 in the upper row of FIG. 2 indicates time, and the vertical axis indicates sound volume (dB). Furthermore, the horizontal axis of the sound data W1 and W2 in the lower row of FIG. 2 indicates time, and the vertical axis indicates frequency (Hz).
[0051] As shown in Figure 2, steady sounds include sounds that occur continuously for a predetermined period of time (e.g., several seconds). For example, steady sounds are sounds that occur steadily for a certain period of time while a manufacturing device is in operation. If the manufacturing device is equipped with a motor, the sound of the motor operating is included in steady sounds.
[0052] The dashed frame in FIG. 2 indicates the range in which a target sound is extracted from sound data. If the sound is a steady sound, the sound is generated continuously for a predetermined period of time, so the range can be easily (e.g., automatically) identified. The target sound is, for example, a sound emitted by a manufacturing device, which is an example of a sound to be extracted. The target sound is also, for example, a sound used to train a machine learning model.
[0053] Fig. 3 is a diagram showing an example of a waveform (sound waveform) of a non-stationary sound. Fig. 3 shows a graph showing the sound volume in sound data W3. The sound data W3 is data obtained by collecting a non-stationary sound. The horizontal axis of Fig. 3 represents time, and the vertical axis represents the sound volume (dB).
[0054] As shown in Fig. 3, non-stationary sounds include multiple sounds that are shorter than a predetermined period (for example, instantaneous sounds). Non-stationary sounds are sounds that are generated locally during operation of a manufacturing device, such as impact sounds or friction sounds. Non-stationary sounds include sounds generated by collisions between objects being processed, or by contact between the objects and a drive mechanism.
[0055] The non-stationary sound includes, for example, multiple alternating first portions W3a (see FIG. 5A described later) with large amplitudes and second portions W3b (see FIG. 5A described later) with small amplitudes. The first portions W3a are sounds collected when the manufacturing equipment is emitting sound. The second portions W3b are sounds collected when the manufacturing equipment is not emitting sound, such as noise or surrounding environmental sounds. For example, if the portion enclosed by the dashed line in FIG. 3 is extracted and used to train a machine learning model, the dashed line portion contains sounds collected when the manufacturing equipment is not emitting sound, which may reduce the accuracy of the generated machine learning model. Therefore, it is desirable to extract a portion of the non-stationary sound to be used for training in the machine learning device 50 (the portion of the target sound, for example, the first portion W3a), but this extraction is not easy. Therefore, the information processing device 10 performs a support process to easily extract the target sound from the non-stationary sound, as described below.
[0056] The acquisition unit 11 acquires sound data obtained by collecting, for example, a stationary sound as shown in FIG. 2 or a non-stationary sound as shown in FIG.
[0057] 1 , the determination unit 12 determines whether or not the sound data acquired by the acquisition unit 11 includes a non-stationary sound. For example, if the sound data includes a portion in which an amplitude equal to or greater than a predetermined value continues for a first time or more, the determination unit 12 determines that the sound data is a stationary sound, and if the sound data includes a portion in which an amplitude equal to or greater than a predetermined value continues for a time less than a second time, the determination unit 12 determines that the sound data is a non-stationary sound. The first time and the second time may be the same, or, for example, the second time may be shorter than the first time.
[0058] The display control unit 13 controls images to be displayed on the display device 30. The display control unit 13 causes the display device 30 to display information related to the sound data acquired by the acquisition unit 11. For example, if the sound data acquired by the acquisition unit 11 includes a non-stationary sound, the display control unit 13 causes the display device 30 to display the waveform of the non-stationary sound. For example, the display control unit 13 generates a control signal for displaying the non-stationary sound included in the sound data acquired by the acquisition unit 11, and outputs the generated control signal to the display device 30, thereby displaying the non-stationary sound on the display device 30. Displaying is an example of presenting. The display control unit 13 is an example of a presentation control unit.
[0059] The sound output control unit 14 controls the sound to be output (played) by the sound output device 40. The sound output control unit 14 causes the sound output device 40 to output sound related to the sound data acquired by the acquisition unit 11. For example, if the sound data acquired by the acquisition unit 11 includes a non-stationary sound, the sound output control unit 14 causes the sound output device 40 to output the non-stationary sound. For example, the sound output control unit 14 generates a control signal for outputting the non-stationary sound included in the sound data acquired by the acquisition unit 11, and outputs the generated control signal to the sound output device 40, thereby causing the sound output device 40 to output the non-stationary sound. Emitting a sound is an example of presenting it. The sound output control unit 14 is an example of a presentation control unit.
[0060] The receiving unit 15 is a user interface that receives input from a user. The receiving unit 15 receives input of a range of sound to be extracted from the non-stationary sound (for example, the range enclosed by a dashed line in FIG. 5B (first target range A1) described later). The receiving unit 15 receives input of a target range (for example, see the first target range A1, etc.), which is a portion to be extracted in the waveform of the non-stationary sound displayed by the display device 30. The receiving unit 15 is realized by a touch panel, a button, a keyboard, etc., but may also be configured to receive input by voice, gestures, etc. The receiving unit 15 is an example of a second acquisition unit.
[0061] The extraction unit 16 extracts, from the waveform of the non-stationary sound, one or more similar ranges (see, for example, a first similar range A11 shown in FIG. 5C , which will be described later) that include waveforms similar to the waveform of the target range inputted via the reception unit 15. Note that the extraction unit 16 may, for example, perform a process of identifying one or more similar ranges to be displayed to the user, but may not necessarily perform a process of extracting the identified one or more similar ranges.
[0062] The processing unit 17 performs predetermined processing on the target range and one or more similar ranges. The processing unit 17 combines the target range and one or more similar ranges. The processing unit 17 may extract the target range and one or more similar ranges from the sound data and combine them. Combining means connecting the waveforms of the target range and one or more similar ranges in time to generate one continuous waveform.
[0063] The storage unit 18 stores the sound data generated by the processing unit 17 as training data for the machine learning model. The storage unit 18 is realized by a semiconductor memory, a hard disk, or the like.
[0064] The evaluation unit 19 evaluates the machine learning model trained by the machine learning device 50 using the training data generated by the information processing device 10. The evaluation unit 19 evaluates the accuracy rate of the output of the machine learning model based on the output obtained by inputting sound data (raw data) containing non-stationary sounds acquired by the acquisition unit 11 into the machine learning model, and on the target range and similar range of the non-stationary sounds.
[0065] The function of the evaluation unit 19 is not limited to being provided in the information processing device 10, but may be provided in the machine learning device 50, for example.
[0066] The sound collection device 20 is disposed near the manufacturing equipment and collects sounds from the manufacturing equipment. The sounds collected by the sound collection device 20 include sounds collected while the manufacturing equipment is in operation. The sound collection device 20 is realized by, for example, a microphone.
[0067] The display device 30 displays various information to the user in accordance with the control of the display control unit 13. The display device 30 is realized by, for example, a liquid crystal display device. The display device 30 may be configured integrally with the information processing device 10.
[0068] The sound output device 40 outputs various sounds to the user in accordance with the control of the sound output control unit 14. The sound output device 40 outputs the sound data acquired by the acquisition unit 11 and the training data generated by the processing unit 17. The sound output device 40 is realized by, for example, a speaker. The sound output device 40 may be configured integrally with the display device 30.
[0069] The machine learning device 50 uses sound data as input data and trains a machine learning model that outputs information indicating the sound emitted by the manufacturing equipment in the sound data. The sound data (training data) registered by the information processing device 10 is used as input data, and the positions on the waveform of the target region and one or more similar regions in the sound data are used as correct answer data.
[0070] The algorithm of the machine learning model may be, for example, a neural network, but the type is not particularly limited. When a neural network is used as the machine learning model, the machine learning device 50 generates the machine learning model by updating network parameters (e.g., weights and biases) of the machine learning model using the sound data registered by the information processing device 10.
[0071] 2. Operation of Information Processing System Next, the operation of the information processing system 1 configured as described above will be described with reference to Fig. 4 to Fig. 7. Fig. 4 is a flowchart showing a sound registration operation (information processing method) in the information processing system 1 according to this embodiment. Fig. 4 shows the operation executed by the information processing device 10 (for example, a computer).
[0072] 4 , the acquisition unit 11 acquires sound data collected from the sound collection device 20 (S10). The timing at which the acquisition unit 11 acquires the sound data is not particularly limited. The acquisition unit 11 outputs the acquired sound data to the determination unit 12.
[0073] Next, the determination unit 12 determines whether the sound data acquired from the acquisition unit 11 includes a non-stationary sound (S20).
[0074] Next, if the determination unit 12 determines that the sound data includes a non-stationary sound (Yes in S20), the display control unit 13 controls the display device 30 to display a waveform of the non-stationary sound on the display device 30, and the sound output control unit 14 controls the sound output device 40 to play (output) the non-stationary sound on the sound output device 40 (S30). For example, the display control unit 13 generates a control signal including a waveform of the non-stationary sound and outputs the generated control signal to the display device 30, thereby causing the display device 30 to display the waveform of the non-stationary sound. Also, for example, the sound output control unit 14 generates a control signal including a waveform of the non-stationary sound and outputs the generated control signal to the sound output device 40, thereby causing the sound output device 40 to play the non-stationary sound.
[0075] In step S30, at least one of displaying the non-stationary sound and playing the non-stationary sound may be performed. Displaying the non-stationary sound and playing the waveform is an example of presenting a waveform. Furthermore, if the determination unit 12 determines Yes in step S20, the determination unit 12 may store sound data W3 (see, for example, FIG. 5A described later) acquired from the sound collection device 20 in the storage unit 18 to use it as evaluation data. The evaluation data stored in the storage unit 18 is, for example, sound data that has not been edited, such as extracted by the extraction unit 16.
[0076] FIG. 5A is a diagram showing collected sound data.
[0077] As shown in Fig. 5A, the display control unit 13 causes the display device 30 to display the waveform of the non-stationary sound. The waveform of the non-stationary sound displayed in step S30 may be raw data that has not been processed from the sound data acquired by the acquisition unit 11. After the waveform shown in Fig. 5A is displayed on the display device 30, the acceptance unit 15 may accept from the user a decision result as to whether or not to register the waveform. When the user is asked to decide whether or not to register the waveform, at least one of displaying the non-stationary sound and playing the non-stationary sound may be executed in step S30.
[0078] Referring again to FIG. 4 , next, the receiving unit 15 acquires a first target range A1 (see FIG. 5B described later) of the waveform to be registered from the non-steady sound (S40). It can also be said that the receiving unit 15 acquires, for example, an input of the first target range A1, which is a portion of the displayed waveform (raw data) to be extracted, from the user. For example, the receiving unit 15 may accept an input of a portion (range) that will become the first target range A1 from the waveform shown in FIG. 5A , or may display multiple candidates for the first target range A1 and accept selection of one or more first target ranges A1 from among them. The first target range A1 is a range of the sound data shown in FIG. 5A that includes waveforms that can be used as sounds for training a machine learning model.
[0079] FIG. 5B is a diagram showing a first target range A1 input for collected sound data.
[0080] 5B, the receiving unit 15 receives, for example, an input of a first target range A1. The first target range A1 is, for example, a range that includes any of the first portions W3a.
[0081] 5B may be displayed on the display device 30 under the control of the display control unit 13. The sound in the first target range A1 shown in FIG. 5B may be output from the sound output device 40 under the control of the sound output control unit 14.
[0082] The receiving unit 15 outputs the received first target range A1 to the extracting unit 16 .
[0083] Referring again to FIG. 4 , the extraction unit 16 extracts one or more first similar ranges A11 similar to the first target range A1 from the waveform shown in FIG. 5A based on the acquired first target range A1 (S50). Any existing technology may be used to extract the first similar range A11. The extraction unit 16 may extract, as the first similar range A11, a range including a waveform similar in shape to the waveform in the first target range A1. Alternatively, the extraction unit 16 may extract, as the first similar range A11, a range including a waveform similar in frequency domain to the waveform in the first target range A1 (e.g., at least partially overlapping) based on a graph showing a frequency spectrogram shown in the lower part of FIG. 2 . Alternatively, the extraction unit 16 may extract, as the first similar range A11, a range including a waveform similar in frequency domain to the waveform in the first target range A1 (e.g., at least partially overlapping). Alternatively, the extraction unit 16 may extract, as the first similar range A11, a range including similar feature quantities based on the waveform in the first target range A1.
[0084] The extraction unit 16 may extract, for example, a range in which the similarity to the waveform in the first target range A1 satisfies a threshold (third threshold) as the first similarity range. The extraction unit 16 may extract, for example, a range in which the similarity to the waveform in the first target range A1 is equal to or greater than the threshold as the first similarity range. The similarity satisfying the threshold may mean, for example, that the correlation coefficient between the two waveforms is equal to or greater than the threshold, that the frequency region of the waveform in the first target range A1 overlaps with the frequency region equal to or greater than the threshold, or that the distance in a two-dimensional space indicating the two-dimensional feature amount into which the waveform in the first target range A1 is transformed is less than the threshold.
[0085] In this way, the extraction unit 16 may automatically extract the first similar range A11 using the similarity of the waveform with the first target range A1. Note that the method by which the extraction unit 16 automatically extracts the first similar range A11 is not limited to this, and any existing technology may be used.
[0086] 5C is a diagram showing one or more first similar ranges A11 automatically extracted from the waveform in the first target range A1. In FIG. 5C, the first similar ranges A11 are indicated by dashed lines.
[0087] FIG. 5C illustrates an example in which three first similar ranges A11 are automatically extracted from the first target range A1. The image illustrated in FIG. 5C may be displayed on the display device 30 under the control of the display control unit 13. For example, the first target range A1 and the first similar range A11 may be simultaneously displayed on one screen. Furthermore, for example, the first target range A1 and the first similar range A11 may be displayed in different modes. The different modes may be, for example, different frame shapes, display colors, or the like. This allows the similar ranges automatically extracted by the extraction unit 16 to be presented to the user. Note that it is sufficient that at least the first similar range A11 is displayed.
[0088] 5C may be output (played) from the sound output device 40 under the control of the sound output control unit 14. For example, the waveforms of the first target range A1 and one or more first similar ranges A11 may be combined by the processing unit 17, and the sound represented by the combined waveform may be output from the sound output device 40 under the control of the sound output control unit 14. This allows the user to hear the sound of the similar range automatically extracted by the extraction unit 16. In this way, at least one of displaying and outputting sound for the similar range allows the user to confirm the automatically extracted similar range.
[0089] In this way, when the user inputs the first target range A1, the extraction unit 16 is configured to automatically extract one or more first similar ranges A11 that are similar to the first target range A1. Note that the first target range A1 and each of the one or more first similar ranges A11 have the same time width (width of the horizontal axis), for example.
[0090] 4 again, the receiving unit 15 determines whether or not a second target range A2 of the waveform has been additionally acquired (for example, see FIG. 5D described later) (S60). The second target range A2 is a range of the displayed waveform to be extracted, and is a range different from the first target range A1. For example, after the image shown in FIG. 5C is displayed, the receiving unit 15 determines whether or not further input of the second target range A2 has been received from the user (for example, whether or not input by the user to a touch panel, button, etc. has been detected).
[0091] FIG. 5D is a diagram showing a second target range A2 input for the collected sound data.
[0092] 5D , the receiving unit 15 receives, for example, an input of a second target range A2. The second target range A2 is, for example, a range that includes any of the first portions W3a and does not overlap with the first target range A1 or the first similar range A11. In FIG. 5D , the second target range A2 is indicated by a two-dot chain line.
[0093] 5D may be displayed on the display device 30 under the control of the display control unit 13. For example, only the second target range A2 may be displayed, or the first target range A1, the first similar range A11, and the second target range A2 may be displayed simultaneously on one screen. Furthermore, the sound of the second target range A2 shown in FIG. 5D may be output from the sound output device 40 under the control of the sound output control unit 14.
[0094] When the receiving unit 15 receives input of the second target range A2 as shown in FIG. 5D, the determination in step S60 is Yes.
[0095] If the extracting unit 16 determines that the receiving unit 15 has received input of the second target range A2 (Yes in S60), the extracting unit 16 further extracts one or more second similar ranges A22 (see, for example, FIG. 5D described later) that are similar to the second target range A2 from the waveform shown in FIG. 5A based on the acquired second target range A2 (S70). The process of extracting the second similar range A22 is similar to the process of extracting the first similar range A11, and therefore, a description thereof will be omitted. If the extracting unit 16 determines that the receiving unit 15 has not received input of the second target range A2 (No in S60), the extracting unit 16 proceeds to step S80.
[0096] 5E is a diagram showing one or more second similar ranges A22 automatically extracted for the waveform in the second target range A2. In FIG. 5E, the second similar ranges A22 are indicated by solid lines.
[0097] FIG. 5E illustrates an example in which three second similar ranges A22 are automatically extracted from the second target range A2. The image illustrated in FIG. 5E may be displayed on the display device 30 under the control of the display control unit 13. For example, the first target range A1, the first similar range A11, the second target range A2, and the second similar range A22 may be simultaneously displayed on one screen. Furthermore, for example, the first target range A1, the first similar range A11, the second target range A2, and the second similar range A22 may be displayed in different modes. This allows the user to view the similar ranges automatically extracted by the extraction unit 16. Note that, in this example, it is sufficient that at least the second similar range A22 is displayed.
[0098] 5E may be output from the sound output device 40 under the control of the sound output control unit 14. For example, the waveforms of the first target range A1, the second target range A2, one or more first similar ranges A11, and one or more second similar ranges A22 may be combined by the processing unit 17, and the sound represented by the combined waveform may be output from the sound output device 40 under the control of the sound output control unit 14. This allows the user to hear the sound of the similar range automatically extracted by the extraction unit 16. In this way, at least one of displaying and outputting sound for the similar range allows the user to confirm the automatically extracted similar range.
[0099] In this way, when the user inputs the second target range A2, the extraction unit 16 automatically extracts one or more second similar ranges A22 that are similar to the second target range A2. Note that the second target range A2 and each of the one or more second similar ranges A22 have the same time width (width of the horizontal axis), for example.
[0100] 4 again, the processing unit 17 combines the extracted similar ranges, and the sound output control unit 14 plays back a sound indicating the combined similar ranges (S80). For example, after step S70, the processing unit 17 extracts waveforms of the first target range A1, one or more first similar ranges A11, the second target range A2, and one or more second similar ranges A22 from the sound data shown in FIG. 5A and combines the extracted waveforms to generate a single waveform. Alternatively, for example, after step S70, the processing unit 17 may extract waveforms of the first target range A1 and one or more first similar ranges A11 from the sound data shown in FIG. 5A and combine the extracted waveforms to generate a single waveform, and further extract waveforms of the second target range A2 and one or more second similar ranges A22 from the sound data shown in FIG. 5A and combine the extracted waveforms to generate a single waveform.
[0101] The sound output control unit 14 causes the sound output device 40 to output a sound indicated by the sound data (training data) combined by the processing unit 17. At this time, the display control unit 13 may cause the display device 30 to display the sound data combined by the processing unit 17.
[0102] Next, after step S80, or when the determination unit 12 determines that the sound data does not contain non-stationary sound (No in S20), the processing unit 17 registers the sound data (S90). That is, the processing unit 17 stores the combined sound data in the storage unit 18 as data for training the machine learning model.
[0103] The registration of sound data may be performed by the reception unit 15 acquiring an input from the user permitting registration after the processing unit 17 has at least one of displayed and / or output the combined sound data to the user. This allows sounds approved by the user to be registered, which can improve the accuracy of analysis using unsteady sounds.
[0104] Next, an operation using the registered sound data (training data) will be described with reference to Fig. 6. Fig. 6 is a flowchart showing a learning operation (information processing method) in the information processing system 1 according to this embodiment.
[0105] 6 , the information processing device 10 trains a machine learning model using the registered sound data (S110). In other words, the machine learning device 50 trains the machine learning model using the sound data registered by the information processing device 10. The information processing device 10 outputs the sound data stored in the storage unit 18 to the machine learning device 50, and the machine learning device 50 acquires the sound data from the information processing device 10 and updates the network parameters of the machine learning model using the acquired sound data. When a predetermined number or more of sound data have been stored in the storage unit 18, the information processing device 10 may output the predetermined number or more of sound data to the machine learning device 50.
[0106] Next, the information processing device 10 determines whether or not the training is completed (S120). For example, the information processing device 10 determines whether or not the learning process in the machine learning device 50 has ended.
[0107] Next, if the information processing device 10 determines that the training is complete (Yes in S120), it evaluates the generated machine learning model (S130), and if it determines that the training is not complete (No in S120), it waits until the training is complete.
[0108] In step S130, the evaluation unit 19 of the information processing device 10 evaluates the machine learning model using sound data for evaluation (evaluation data). The evaluation data may include, for example, the sound data acquired in step S10 of Fig. 4. The evaluation data is stored in, for example, the storage unit 18.
[0109] 7 is a diagram illustrating a method for evaluating a generated machine learning model according to this embodiment. (a) of FIG. 7 shows evaluation sound data W3 input to the machine learning model, which is, for example, sound data W3 containing non-stationary sound acquired in step S10 of FIG. 4. In other words, the original sound data (raw data) of the sound data used to train the machine learning model is used as the evaluation data. The evaluation data may be preset sound data dedicated to evaluation.
[0110] 7B shows a registered model (trained machine learning model) stored in the memory unit 51 of the machine learning device 50. FIG. 7C shows an output image from the machine learning model. The horizontal axis of the graph in FIG. 7C represents time, and the vertical axis represents High (e.g., "1") and Low (e.g., "0"). The output from the machine learning model is High (e.g., "1") when the target sound is emitted from the input sound data, and Low (e.g., "0"). In this way, the machine learning model is a mathematical model that inputs a non-stationary sound including the first portion W3a and the second portion W3b and outputs the portion of the target sound emitted by the manufacturing equipment as High. FIG. 7C shows an output image when the target sound is included eight times in the input sound data.
[0111] The evaluation unit 19 evaluates the machine learning model based on at least one of the number of times a target sound is emitted in the input sound data and the time positions at which the target sound is emitted, and at least one of the number of times the output of the machine learning model becomes High (e.g., "1" on the vertical axis) and the time positions at which the output becomes High (e.g., "1" on the vertical axis). For example, the evaluation unit 19 may give the highest evaluation when the output of the machine learning model is High at each of multiple time positions at which the target sound is emitted in the input sound data (i.e., when the number of sounds and the time positions match), and may lower the evaluation when the output of the machine learning model corresponding to some of the multiple target sounds included in the input sound data is Low, or when the output of the machine learning model corresponding to a sound other than the target sound included in the input sound data (e.g., the sound of the second part W3b) is High. The evaluation unit 19 outputs, for example, accuracy as the evaluation result. The accuracy is, for example, a numerical value ranging from 0 to 100. The evaluation result may be, for example, a correct answer rate or an incorrect answer rate.
[0112] 6 again, the evaluation unit 19 then determines whether the accuracy obtained as the evaluation result is greater than a predetermined value (S140). The predetermined value is a threshold value for determining whether additional training is required for the machine learning model, and is set in advance and stored in the storage unit 18.
[0113] If the accuracy is greater than a predetermined value (Yes in S140), the evaluation unit 19 terminates the learning process, and if the accuracy is equal to or less than the predetermined value (No in S140), the evaluation unit 19 executes a re-learning process (S150). The re-learning process is, for example, a process of generating additional training data for the machine learning model and re-updating the network parameters of the machine learning model.
[0114] The machine learning model generated in this way is used to understand the operating status of the manufacturing equipment. For example, by counting the number of times the machine learning model outputs a High value, it is possible to understand the number of times the manufacturing equipment has operated.
[0115] (Variation 1 of the embodiment) The information processing method according to this variation will be described below with reference to FIG. 8. The following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified. The configuration of the information processing system may be the same as the information processing system 1 of the embodiment, and description of the configuration of the information processing system according to this variation will be omitted. The following description will use the reference numerals of the information processing system 1 used in the embodiment.
[0116] 8 is a flowchart showing a sound registration operation (information processing method) in the information processing system 1 according to this modification. The information processing method according to this modification differs from the information processing method according to the embodiment in that the operations of steps S210 to S260 are further executed after a No determination is made in step S60 shown in FIG. 6 or between step S70 and step S80. Note that, for convenience, the following describes the operation when a No determination is made in step S60, but the same applies when the operation is executed after step S70.
[0117] 8 , the processing unit 17 determines whether the variation of one or more first similar ranges A11 is less than a first threshold (S210). The processing unit 17 calculates the variation based on the similarity between the first target range A1 and each of the one or more first similar ranges A11. For example, the processing unit 17 calculates the variation as the standard deviation of the similarity between the first target range A1 and each of the one or more first similar ranges A11, but this is not limiting.
[0118] Note that a Yes judgment in step S120 corresponds to, for example, the variation of one or more first similarity ranges A11 based on the similarity (first similarity) between the waveform of the first target range A1 and each waveform of one or more first similarity ranges A11 being smaller than a first threshold value.
[0119] Next, if the processing unit 17 determines that the variation is less than the first threshold (Yes in S210), the extraction unit 16 newly sets a third target range including waveforms whose similarity to the waveform in the first target range A1 differs from the similarity of one or more of the first similar ranges A11 extracted in step S50, and the display control unit 13 causes the display device 30 to display the extracted third target range (S220). The extraction unit 16 automatically extracts the third target range. It can also be said that the extraction unit 16 identifies a third target range whose similarity to the waveform in the first target range A1 differs from the first similarity. For example, if the third threshold for determining similarity is 80 and the first similarity is 95, the extraction unit 16 may identify a range in which the similarity is 80 to 90 as the third target range. For example, if the first similarity is 85, the extraction unit 16 may identify a range in which the similarity is 90 to 100 as the third target range. In this way, the extraction unit 16 identifies a range of similarity that satisfies the threshold and is different from the first similarity as the third target range. The first similarity here is the average value of the similarities of one or more first similarity ranges A11, but may also be the median, mode, representative value, minimum value, maximum value, or the like.
[0120] Next, the receiving unit 15 determines whether or not an instruction to add the third target range displayed in step S220 has been acquired (S230). If the receiving unit 15 receives an input from the user to add the third target range (for example, if the receiving unit 15 detects an input from the user to a touch panel, button, or the like), the determination in step S230 is Yes. Note that the processing in step S230 may be omitted.
[0121] Next, when the receiving unit 15 receives an instruction to add a third target range (Yes in S230), the extracting unit 16 extracts a similar range (an example of a second similar range) that is similar to the third target range (S240). The method for extracting a similar range that is similar to the third target range may be the same as that in step S50 shown in FIG. 4, and therefore, a description thereof will be omitted.
[0122] The waveform in the similar range extracted in step S240 is considered to be a waveform that is less similar to the target sound (e.g., the sound with a waveform in the first target range A1) than the waveform in the similar range extracted in step S50. In step S240, a range that includes a waveform that is slightly distorted from the target sound can be extracted as the similar range.
[0123] Furthermore, when the processing unit 17 determines that the variation is equal to or greater than the first threshold (No in S210), it further determines whether the variation is greater than a second threshold that is greater than the first threshold (S250). When the processing unit 17 determines that the variation is greater than the second threshold (Yes in S250), it changes the similarity threshold (third threshold) for determining similarity (S260). When the processing unit 17 determines that the variation is greater than the second threshold, it changes the third threshold to a fourth threshold that is higher than the third threshold.
[0124] Next, the extraction unit 16 extracts one or more first similar ranges A11 again using the fourth threshold changed by the processing unit 17 (S270). For example, the extraction unit 16 excludes first similar ranges A11 that do not satisfy the changed fourth threshold from the one or more first similar ranges A11 extracted in step S50. Then, the process proceeds to step S250.
[0125] In addition, if the receiving unit 15 does not receive an instruction to add a third target range (No in S230), or if the processing unit 17 determines that the variation is less than the second threshold (i.e., greater than or equal to the first threshold and less than the second threshold) (No in S250), the information processing device 10 proceeds to step S80.
[0126] As a result, when the variation in similarity is small, the similar range is re-extracted so that the variation in similarity is large. By training a machine learning model using sound data including a similar range extracted so that the similarity varies to a certain extent in this way, a more versatile machine learning model can be generated. Furthermore, when the variation in similarity is large, a similar range is selected so that the variation in similarity is small, so that a similar range including a sound other than the target, such as noise, can be excluded. By training a machine learning model using sound data including a similar range selected so that the similarity is small in this way, a more accurate machine learning model can be generated. Furthermore, by omitting the processing of step S230, the variation in the similarity range can be automatically adjusted, so the information processing device 10 can effectively assist in extracting a target sound from non-stationary sounds.
[0127] The first threshold value and the second threshold value may be the same value.
[0128] (Variation 2 of the Embodiment) The information processing method according to this variation will be described below with reference to FIG. 9 . The following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified. The configuration of the information processing system may be the same as the information processing system 1 of the embodiment, and description of the configuration of the information processing system according to this variation will be omitted. The following description will use the reference numerals of the information processing system 1 used in the embodiment.
[0129] 9 is a flowchart showing a sound registration operation (information processing method) in the information processing system 1 according to this modification. The information processing method according to this modification differs from the information processing method according to the embodiment in that the operations of steps S310 to S360 are further executed after a No determination is made in step S60 shown in FIG. 6 or between step S70 and step S80. Note that, for convenience, the following describes the operation when a No determination is made in step S60, but the same applies when the operation is executed after step S70.
[0130] As shown in FIG. 9, the processing unit 17 determines whether the number of extracted first similar ranges A11 is less than a fifth threshold (S310).
[0131] Next, when the processing unit 17 determines that the number of extractions is less than the fifth threshold (Yes in S310), the extraction unit 16 further sets a fourth target range, and the display control unit 13 causes the display device 30 to display the extracted fourth target range (S320). The fourth target range is a range that does not overlap with the first target range A1 and one or more first similar ranges A11, and may be automatically extracted by the extraction unit 16 based on the similarity.
[0132] Next, the accepting unit 15 determines whether or not an instruction to add the fourth target range displayed in step S320 has been received from the user (S330). If the accepting unit 15 has received an input from the user to add the fourth target range (for example, if the accepting unit 15 has detected an input from the user to a touch panel, button, or the like), the accepting unit 15 determines Yes in step S330. Note that the processing of step S330 may be omitted.
[0133] Next, when the receiving unit 15 receives an instruction to add a fourth target range (Yes in S330), the extracting unit 16 extracts a similar range (an example of a second similar range) that is similar to the fourth target range (S340). The method for extracting a similar range that is similar to the fourth target range may be the same as that in step S50 shown in FIG. 4, and therefore, a description thereof will be omitted.
[0134] Furthermore, when the processing unit 17 determines that the number of extracted images is equal to or greater than the fifth threshold (No in S310), it further determines whether the number of extracted images is greater than a sixth threshold that is greater than the fifth threshold (S350). When the processing unit 17 determines that the number of extracted images is greater than the sixth threshold (Yes in S350), it changes the similarity threshold (third threshold) for determining similarity (S360). When the processing unit 17 determines that the number of extracted images is greater than the sixth threshold (Yes in S350), it changes the third threshold to a fourth threshold that is higher than the third threshold.
[0135] Next, the extraction unit 16 extracts one or more first similar ranges A11 again using the fourth threshold changed by the processing unit 17 (S370). For example, the extraction unit 16 excludes any first similar ranges A11 that do not satisfy the changed fourth threshold from the one or more first similar ranges A11 extracted in step S50. Then, the process proceeds to step S350.
[0136] In addition, if the receiving unit 15 does not receive an instruction to add a fourth target range (No in S330), or if the processing unit 17 determines that the number of extractions is less than the sixth threshold (i.e., greater than or equal to the fifth threshold and less than the sixth threshold) (No in S350), the information processing device 10 proceeds to step S80.
[0137] As a result, if the number of extracted similar ranges is small, additional target ranges are set to increase the number of extracted similar ranges, and if the number of extracted similar ranges is large, the similarity threshold (third threshold) is changed to reduce the number of extracted similar ranges, so that the number of similar ranges can be set to a predetermined number. By training a machine learning model using sound data including a number of similar ranges within such a predetermined range, it is possible to generate a machine learning model with higher accuracy while suppressing an increase in the processing load of the machine learning device 50. Furthermore, by omitting the processing of step S330, the number of extracted similar ranges can be automatically adjusted, thereby effectively supporting the extraction of target sounds from non-stationary sounds.
[0138] The fifth threshold and the sixth threshold may be the same value.
[0139] (Variation 3 of the embodiment) The information processing method according to this variation will be described below with reference to FIGS. 10 to 11B. Note that the following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified. Note that the configuration of the information processing system may be the same as the information processing system 1 of the embodiment, and description of the configuration of the information processing system according to this variation will be omitted. Furthermore, the following description will use the reference numerals of the information processing system 1 used in the embodiment.
[0140] 10 is a flowchart showing a sound registration operation (information processing method) in the information processing system 1 according to this modification. The information processing method according to this modification differs from the information processing method according to the embodiment in that steps S410 to S440 are executed instead of steps S40 to S80 shown in FIG.
[0141] 10 , after the waveform is displayed and the sound is played (S30), the extraction unit 16 automatically extracts a range of registration candidates from the sound data W3 (S410). That is, the extraction unit 16 automatically extracts a range of registration candidates from the sound data W3 without acquiring the first target range after step S30.
[0142] Any existing technology may be used to extract the range of registration candidates. The extraction unit 16 may extract, from the sound data W3, a range including waveforms similar in shape to a pre-registered target waveform as the range of registration candidates. Furthermore, the extraction unit 16 may extract, from the sound data W3, a range including waveforms whose frequency domains are similar to (e.g., at least partially overlap with) the pre-registered target waveform based on a graph showing a frequency spectrogram, as the range of registration candidates. Furthermore, the extraction unit 16 may extract, from the sound data W3, a range including similar features based on the pre-registered target waveform as the range of registration candidates. In this way, the extraction unit 16 may automatically extract the range of registration candidates using the similarity to the pre-registered target waveform. Note that the method by which the extraction unit 16 automatically extracts the range of registration candidates is not limited to this, and any existing technology may be used.
[0143] FIG. 11A is a diagram showing a range of registration candidates A31 to A36 automatically extracted from collected sound data.
[0144] 11A shows an example in which ranges A31 to A36 of registration candidates are extracted from sound data W3 by automatic extraction. Note that the number of ranges of registration candidates that are automatically extracted is not particularly limited, and may be one or more.
[0145] 10 again, next, the sound output control unit 14 controls the sound output device 40 to play (output) sounds of the automatically extracted range A31 to A36 of registration candidates (S420). For example, the processing unit 17 combines the automatically extracted range A31 to A36 of registration candidates, and the sound output control unit 14 plays sounds indicating the combined range A31 to A36 of registration candidates. At this time, the display control unit 13 may display the sound data of the range A31 to A36 of registration candidates combined by the processing unit 17 on the display device 30.
[0146] Next, the accepting unit 15 determines whether or not a change by the user to the automatically extracted range A31 to A36 of registration candidates has been accepted (S430). If the accepting unit 15 has accepted an input from the user to change the range A31 to A36 of registration candidates (for example, if the accepting unit 15 has detected an input by the user to a touch panel, button, or the like), the answer to step S430 is Yes.
[0147] Next, if the accepting unit 15 determines that the change has been accepted (Yes in S430), the processing unit 17 reflects the change made by the user (S440). The change includes deleting part of the extracted range of registration candidates A31 to A36. For example, the user can check whether there are any errors in the automatically extracted range of registration candidates and delete any errors if any. Note that the change may involve adding an additional range of registration candidates to the automatically extracted range of registration candidates A31 to A36, or may involve changing the size (for example, the horizontal width) of the range of registration candidates.
[0148] FIG. 11B is a diagram showing the range of registration candidates after a part of the range of registration candidates has been deleted by the user.
[0149] 11B shows an example in which the user has deleted registration candidate range A34 from the automatically extracted registration candidate range A31 to A36. In this case, in step S90, registration candidate ranges A31 to A33 and registration candidate ranges A35 and A36 from the registration candidate range A31 to A36 are stored in the storage unit 18 as training data for the machine learning model.
[0150] If the accuracy of the automatic extraction is equal to or higher than a predetermined level, the processes of steps S430 and S440 may be omitted.
[0151] As described above, by automatically detecting registration candidates for non-stationary sounds, the burden on the user to select registration candidates can be reduced, particularly in the case of long sound data.
[0152] (Other Embodiments) As above, information processing systems etc. according to one or more aspects have been described based on embodiments etc., but the present disclosure is not limited to these embodiments etc. As long as they do not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art can conceive of to the present embodiments and embodiments constructed by combining components of different embodiments may also be included in the present disclosure.
[0153] For example, in the above embodiments, a manufacturing device is exemplified as an object that emits sound, but the object may be an image forming device having a copy function, a printer function, etc., an air conditioning device, etc. The object may be, for example, any device that has one or more drive mechanisms.
[0154] In addition, in the above embodiments, the object is described as a device that emits either a steady sound or a non-steady sound, but the object is not limited to this and may be a device that emits a mixture of steady sound and non-steady sound. The information processing method and the like according to the present disclosure are also effective for such a device.
[0155] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.
[0156] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.
[0157] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.
[0158] Furthermore, each of the information processing devices or machine learning devices according to the above embodiments may be realized as a single device or multiple devices. When an information processing device or machine learning device is realized as multiple devices, the components of the information processing device or machine learning device may be distributed among the multiple devices in any manner. When realized as multiple devices, the communication method between the multiple devices is not particularly limited and may be wireless communication or wired communication. Wireless communication and wired communication may be combined between the devices. Furthermore, some or all of the functions of one of the information processing device and the machine learning device may be provided in the other device of the information processing device and the machine learning device. For example, the information processing device and the machine learning device may be realized as an integrated device.
[0159] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, which is an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Here, the term "LSI" is used, but depending on the level of integration, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. After LSI fabrication, a field programmable gate array (FPGA) that can be programmed or a reconfigurable processor that can reconfigure the connections or settings of circuit cells within the LSI may also be used. Furthermore, if an integrated circuit implementation technology that replaces LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.
[0160] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip. Specifically, it is a computer system that includes a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), etc. Computer programs are stored in the ROM. The system LSI achieves its functions when the microprocessor operates in accordance with the computer program.
[0161] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the information processing method shown in any of Figures 4, 6, 8, 9, and 10.
[0162] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes.
[0163] The present disclosure is useful for an information processing device or the like that processes sound data obtained by collecting sounds emitted by an object.
[0164] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device 11 Acquisition unit (first acquisition unit) 12 Determination unit 13 Display control unit (presentation control unit) 14 Sound output control unit (presentation control unit) 15 Reception unit (second acquisition unit) 16 Extraction unit 17 Processing unit 18, 51 Storage unit 19 Evaluation unit 20 Sound collection device 30 Display device 40 Sound output device 50 Machine learning device A1 First target range A2 Second target range A11 First similarity range A22 Second similarity range A31, A32, A33, A34, A35, A36 Range of registration candidates W1, W2, W3 Sound data W3a First part W3b Second part
Claims
1. An information processing method executed by a computer, comprising: acquiring sound data obtained by collecting sound from an object; if the sound data includes a non-stationary sound, presenting a waveform of the non-stationary sound; acquiring an input of a first target range which is a range to be extracted from the presented waveform; and extracting one or more first similar ranges which include waveforms similar to the acquired waveform in the first target range from the sound data.
2. The information processing method of claim 1, further comprising: after extracting the one or more first similar ranges, acquiring an input of a second target range, which is another range to be extracted in the presented waveform; and extracting one or more second similar ranges from the sound data, the second similar range including waveforms similar to the acquired waveform of the second target range.
3. The information processing method of claim 1, further comprising: identifying a second target range having a different similarity to the waveform in the first target range from the first similarity when the variation of the one or more first similarity ranges based on a first similarity between the waveform in the first target range and each of the waveforms in the one or more first similarity ranges is smaller than a first threshold; and extracting one or more second similarity ranges from the sound data that include waveforms similar to the identified waveform in the second target range.
4. The information processing method of claim 1, wherein the one or more first similarity ranges are ranges including waveforms having a similarity to the waveform in the first target range that is equal to or greater than a third threshold, and when a variation in the one or more first similarity ranges based on a first similarity between the waveform in the first target range and each of the waveforms in the one or more first similarity ranges is greater than a second threshold, the third threshold is changed to a fourth threshold that is greater than the third threshold, and the one or more first similarity ranges are re-extracted based on the changed fourth threshold.
5. The information processing method of claim 1, further comprising: if the number of the one or more extracted first similar ranges is smaller than a fifth threshold, identifying a second target range different from the first target range and the one or more first similar ranges; and extracting one or more second similar ranges from the sound data, the second similar ranges including waveforms similar to the waveforms of the identified second target ranges.
6. The information processing method of claim 1, wherein the one or more first similarity ranges are ranges that include waveforms having a similarity to the waveforms in the first target range that is equal to or greater than a third threshold, and when the number of the one or more extracted first similarity ranges is greater than a sixth threshold, the third threshold is changed to a fourth threshold that is greater than the third threshold, and the one or more first similarity ranges are re-extracted based on the changed fourth threshold.
7. The information processing method according to claim 2, further comprising: presenting the extracted one or more first similarity ranges; and after presenting the one or more first similarity ranges, obtaining the input of the second target range.
8. The information processing method according to claim 2, 3 or 5, further comprising presenting the extracted one or more second similarity ranges.
9. The information processing method according to any one of claims 1 to 6, further comprising: displaying a waveform of the sound data and playing back the sound data; and obtaining the input of the first target range after displaying and playing back the sound data.
10. An information processing method according to any one of claims 1 to 6, further comprising the steps of: combining the waveform in the first target range and the waveform in one or more first similar ranges; and playing back sound data including the combined waveform.
11. An information processing method according to any one of claims 2, 3 and 5, comprising combining waveforms of the first target range, the second target range, the one or more first similar ranges and the one or more second similar ranges, and playing sound data including the combined waveform.
12. An information processing method according to any one of claims 1 to 6, wherein the object is a manufacturing device, and the sound of the object includes sound captured while the manufacturing device is in operation.
13. An information processing device comprising: a first acquisition unit that acquires sound data obtained by collecting sound from an object; a presentation control unit that, if the sound data includes a non-stationary sound, presents a waveform of the non-stationary sound; a second acquisition unit that acquires an input of a first target range that is a range to be extracted from the presented waveform; and an extraction unit that extracts one or more first similar ranges from the sound data that include waveforms similar to the acquired waveform of the first target range.
14. A program for causing a computer to execute the information processing method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Stationary section determining method and segmentation device for voiced sound
JP1999305788A
Device and method for discriminating sound source
JP2004264292A
Unusual sound determination device, unusual sound determination method, and unusual sound determination system
JP2020183904A
