Information processing method, information processing device, sound registration system, and program
The information processing method and device simplify sound registration by clustering sound features and providing visual feedback, addressing inefficiencies in existing sound extraction methods and improving the accuracy of machine status estimation in facilities.
Patent Information
- Application Number
- PCT/JP2024/016954
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-21
AI Technical Summary
Existing methods for obtaining appropriate operating sounds for machine status estimation in facilities like manufacturing factories and distribution warehouses are inefficient due to the complexity of sound extraction and difficulty in accurately extracting sound features, especially when sounds from the same action vary slightly each time it is repeated.
An information processing method and device that acquire sound-related information, calculate feature vectors, cluster sounds into groups based on these vectors, and output information for user selection, allowing for easier identification and registration of target sounds by visualizing temporal changes and characteristics, enabling more accurate sound extraction and registration.
This approach simplifies the sound registration process by allowing users to select appropriate sounds for registration based on visual information, reducing the complexity of sound extraction and improving the accuracy of sound feature identification, leading to better machine status estimation.
Smart Images

Figure JP2024016954_21112024_PF_FP_ABST
Abstract
Description
Information processing method, information processing device, sound registration system, and program
[0001] The present disclosure relates to an information processing method, an information processing device, a sound registration system, and a program for registering a sound.
[0002] Conventionally, there has been a demand for managing the operational status of machines in facilities that utilize machines for work, such as manufacturing plants and logistics warehouses. Such operational status management can be achieved using the operating sounds of the machines. Meanwhile, to estimate the operational status of a machine from its operating sounds, it is necessary to obtain appropriate operational sounds and associate them with the operational status (an example of registering the sounds). As a means for obtaining appropriate operational sounds, for example, a sound extraction process, as disclosed in Patent Literature 1, can be used, which removes noise other than that of the machine from input sounds, including sounds emitted by the machine, and selectively extracts the target machine sounds.
[0003] In Patent Document 1, sound is collected from multiple positions, and the feature of the target sound is identified based on the feature of the sound and the distance of the sound from the sound collector, and the feature of the sound is extracted.
[0004] Japanese Patent Application Laid-Open No. 2021-71586
[0005] However, there is room for improvement in using the sound extraction process disclosed in Patent Document 1 as a means for obtaining appropriate operation sounds.
[0006] Therefore, an object of the present disclosure is to provide an information processing method and the like that can obtain a more appropriate sound to be used for registration.
[0007] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer for assisting in the registration of a sound as a reference sound in a sound discrimination device that discriminates sounds by learning the reference sound, and includes the steps of acquiring sound-related information that includes the sound and represents at least one of a change in the occurrence situation of the sound over time or a change in the characteristics of the sound over time; calculating a feature vector of the sound included in the sound-related information; clustering the sounds included in the sound-related information into two or more clusters based on the feature vector, and creating cluster information that indicates to which of the two or more clusters the sounds belong; and creating and outputting information that includes the sound-related information and the cluster information.
[0008] Furthermore, an information processing device according to one aspect of the present disclosure is an information processing device for assisting in the registration of a sound as a reference sound in a sound discrimination device that discriminates sounds by comparing them with the reference sound, and includes an acquisition unit that acquires sound-related information that includes the sound and represents at least one of a change in the occurrence status of the sound over time or a change in the characteristics of the sound over time; a calculation unit that calculates a feature vector of the sound included in the sound-related information; a creation unit that clusters the sounds included in the sound-related information into two or more clusters based on the feature vector and creates cluster information that indicates to which of the two or more clusters the sounds belong; and an output unit that creates and outputs information that includes the sound-related information and the cluster information.
[0009] a calculation unit that calculates a feature vector of the sound included in the sound-related information; a creation unit that clusters the sounds included in the sound-related information into two or more clusters based on the feature vectors and creates cluster information indicating to which of the two or more clusters the sounds belong; an output unit that creates and outputs information including the sound-related information and the cluster information; a detection unit that detects selection of a portion of the time domain of the sound-related information as a registration target range in response to display of the output information; and a registration unit that registers sounds included in the sound-related information that fall within the registration target range as sounds for registration, by outputting the sounds separated from sounds that do not fall within the registration target range.
[0010] Furthermore, one aspect of the present disclosure can also be realized as a program for causing a computer to execute the information processing method described above.
[0011] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.
[0012] According to the present disclosure, it is possible to obtain a more appropriate sound.
[0013] FIG. 1 is a schematic diagram showing a use case of a sound registration system according to embodiment 1. FIG. 2 is a block diagram showing the functional configuration of the sound registration system according to embodiment 1. FIG. 3 is a flowchart showing an example of the operation of the sound registration system according to embodiment 1. FIG. 4 is a diagram showing an example of a screen displayed on a terminal device in the sound registration system according to embodiment 1. FIG. 5 is a block diagram showing the functional configuration of a calculation unit of an information processing device according to embodiment 2. FIG. 6 is a flowchart showing an example of the operation of the sound registration system according to embodiment 2. FIG. 7 is a diagram showing an example of a screen displayed on a terminal device in the sound registration system according to embodiment 2. FIG. 8 is a diagram partially showing an example of a screen displayed on a terminal device in a sound registration system according to another example of an embodiment. FIG. 9 is a diagram partially showing an example of a screen displayed on a terminal device in a sound registration system according to yet another example of an embodiment.
[0014] (Knowledge that forms the basis of the disclosure) In the past, there has been a demand for managing the operational status of machines in facilities that utilize machines for work, such as manufacturing plants and logistics warehouses. Such operational status management involves, for example, estimating the corresponding operational status using the waveform of the power supplied from the machine's power source, log information such as the program that controls each part of the machine, and physical quantities detected by an external sensor. In particular, using physical quantities detected by an external sensor is useful because it enables estimation of the operational status by retrofitting the external sensor even if the machine itself does not have a configuration for estimating the operational status. A known technique for detecting physical quantities detected by such external sensors is to collect the operating sound of a machine using a sound collector. Specifically, by distinguishing between the operating sound collected by the sound collector when the machine performs a certain operation and the operating sound collected by the sound collector when the machine performs a different operation, it is possible to estimate whether the machine is currently performing a certain operation or a different operation.
[0015] On the other hand, to estimate the operating status of a machine from its operating sound as described above, it is necessary to obtain appropriate operating sounds and associate them with the operating status (an example of registering the sounds). As a means for obtaining appropriate operating sounds, for example, a sound extraction process, as disclosed in Patent Document 1, can be used, which removes noise other than that of the machine from input sounds, including sounds emitted by the machine, and selectively extracts the target machine sound. In Patent Document 1, sounds are collected from multiple positions, and the target sound feature is identified based on the sound feature and the distance of the sound from a sound collector, and the sound feature is extracted. However, such a sound extraction process requires collecting sounds from multiple positions, which is cumbersome. Furthermore, there is a risk that accurate sound feature extraction will not be possible due to slight differences in the operating sound that occur with repeated repetitions of the same operation, or that it will be difficult to set a threshold value for the difference to allow for such differences in operating sound.
[0016] Therefore, the present disclosure provides a system configured to receive input from a user to obtain appropriate operation sounds (hereinafter sometimes simply referred to as sounds) and to allow the user to specify a specific appropriate operation sound (i.e., a target sound) from among consecutive sounds in a time series, as well as an information processing method executed in the system. Furthermore, sounds contained in a video capturing the machine's operating status are used as pre-extraction sounds, including the operation sounds. As a result, while viewing sound-related information such as a video corresponding to the sound, the user can easily extract portions where the target operation is definitely occurring or exclude portions where the target operation is clearly not occurring from the extraction, based on visual information. This has the advantage of facilitating user input of information into the system. Examples of using video image information in addition will also be described as other embodiments.
[0017] (Summary of the Disclosure) The summary of the disclosure will be listed below.
[0018] An information processing method according to a first aspect of the present disclosure is an information processing method executed by a computer for assisting in the registration of a sound as a reference sound in a sound discrimination device that discriminates sounds by learning reference sounds, and includes the steps of acquiring sound-related information that includes the sound and represents at least one of a change in the occurrence situation of the sound over time or a change in the characteristics of the sound over time; calculating a feature vector of the sound included in the sound-related information; clustering the sounds included in the sound-related information into two or more clusters based on the feature vector and creating cluster information that indicates to which of the two or more clusters the sounds belong; and creating and outputting information that includes the sound-related information and the cluster information.
[0019] In this information processing method, a user can select a registration target range by receiving display information including sound-related information and cluster information. The cluster information allows the user to determine whether a sound is a target sound based on a set of sounds that are clustered into a single cluster as similar sounds. For example, even if there is a section in which similar sounds are consecutive in time series, the operation of finding the start and end points is cumbersome. However, in this case, because sounds of the same type are clustered into a single cluster, finding the start and end points is not necessary, and the user only needs to determine which cluster is the target sound. Therefore, it is possible to obtain sounds suitable for registration from a registration target range that the user has determined to be appropriate from the perspective of usability.
[0020] Furthermore, an information processing method according to a second aspect of the present disclosure is the information processing method described in the first aspect, wherein the information includes, as sound-related information, a timeline on which sound occurrence conditions at each of multiple points in time that constitute the sound-related information or sound characteristics are arranged in chronological order, and, as cluster information, markers that are attached to corresponding positions in the chronological order on the timeline, and the display manner of the markers differs depending on which of two or more clusters the markers belong to.
[0021] This allows the user to view the cluster information as a marker on the timeline of the sound-related information. By using this marker, the user can associate the sound-related information with the cluster information, making it easier to select an appropriate range to register.
[0022] Furthermore, an information processing method according to a third aspect of the present disclosure is the information processing method described in the first or second aspect, wherein in the step of calculating a feature vector, a first-dimensional vector having features of a first number of dimensions is calculated from sounds included in the sound-related information, and a feature vector having features of a number of dimensions less than the first number of dimensions is calculated by converting the calculated first-dimensional vector, and in the step of creating cluster information, the sounds included in the sound-related information are clustered into two or more clusters based on the feature vector having features of a number of dimensions less than the first number of dimensions.
[0023] This allows the number of dimensions of the feature vector to be reduced from the first number of dimensions of the calculated first-dimensional vector to a number of dimensions that is smaller than the first number of dimensions, which has the advantage of reducing the amount of processing required for clustering compared to when the first-dimensional vector is directly subjected to clustering.
[0024] Furthermore, an information processing method according to a fourth aspect of the present disclosure is the information processing method described in any one of the first to third aspects, wherein the step of calculating a feature vector calculates a first-dimensional vector having features of a first dimension from sounds included in the sound-related information, and calculates a feature vector having features with two dimensions, which is less than the first dimension, by converting the calculated first-dimensional vector; the step of creating cluster information further includes a step of clustering the sounds included in the sound-related information into two or more clusters based on the feature vectors having features with two dimensions, and creating two-dimensional data in which the similarities between each of the feature vectors are expressed as two-dimensional distances between each of the feature vectors; and the information further includes the created two-dimensional data.
[0025] This allows the display information to include two-dimensional data together with sound-related information and cluster information. The user can select a registration target range by relying on the two-dimensional data, which is not possible with cluster information. That is, this has the advantage of enabling the selection of an appropriate registration target range even in a situation where an inappropriate registration target range would be selected based on the cluster information.
[0026] Furthermore, an information processing method according to a fifth aspect of the present disclosure is an information processing method according to any one of the first to fourth aspects, wherein the feature vectors are a plurality of vectors corresponding to unit sounds obtained by dividing sounds included in the sound-related information into unit times.
[0027] This makes it possible to create cluster information for feature vectors, which are a plurality of vectors corresponding to unit sounds obtained by dividing sounds included in the sound-related information into unit times.
[0028] An information processing method according to a sixth aspect of the present disclosure is the information processing method according to any one of the first to fifth aspects, wherein the cluster information included in the information allows feature vectors belonging to one of two or more clusters to be distinguished from feature vectors belonging to other clusters by difference in display color.
[0029] This allows the feature vectors of each cluster to be distinguished from one another by the difference in display color.
[0030] An information processing method according to a seventh aspect of the present disclosure is an information processing method according to any one of the first to sixth aspects, wherein the cluster information included in the information makes it possible to distinguish between feature vectors belonging to one cluster and feature vectors belonging to another cluster among two or more clusters by differences in display shape.
[0031] This allows the feature vectors of each cluster to be distinguished from one another by the difference in display shape.
[0032] In addition, an information processing method according to an eighth aspect of the present disclosure is an information processing method described in any one of the first to seventh aspects, and further includes a step of detecting a selection of a part of the time domain of the sound-related information as a registration target range in response to the display of output information, and a step of outputting sounds included in the sound-related information that fall within the registration target range, separating them from sounds that do not fall within the registration target range, and registering the sounds that fall within the registration target range as sounds for registration.
[0033] According to this, among the sounds included in the sound-related information, sounds that fall within the range to be registered can be used as sounds for registration, and can be output separately from sounds that do not fall within the range to be registered, and used for registration.
[0034] Furthermore, an information processing device according to a ninth aspect of the present disclosure is an information processing device for assisting in the registration of a sound as a reference sound in a sound discrimination device that discriminates sounds by comparing them with a reference sound, and includes an acquisition unit that acquires sound-related information that includes the sound and represents at least one of a change in the occurrence situation of the sound over time or a change in the characteristics of the sound over time; a calculation unit that calculates a feature vector of the sound included in the sound-related information; a creation unit that clusters the sounds included in the sound-related information into two or more clusters based on the feature vector and creates cluster information that indicates to which of the two or more clusters the sounds belong; and an output unit that creates and outputs information including the sound-related information and the cluster information.
[0035] This can achieve the same effect as the above-described information processing method.
[0036] a calculation unit that calculates a feature vector of the sound included in the sound-related information; a creation unit that clusters the sounds included in the sound-related information into two or more clusters based on the feature vectors and creates cluster information indicating to which of the two or more clusters the sounds belong; an output unit that creates and outputs information including the sound-related information and the cluster information; a detection unit that detects selection of a portion of the time domain of the sound-related information as a registration target range in response to display of the output information; and a registration unit that registers sounds included in the sound-related information that fall within the registration target range as sounds for registration, by outputting the sounds separately from sounds that do not fall within the registration target range.
[0037] This makes it possible to achieve the same effect as the above-described information processing method and to register the sound in the sound discrimination device.
[0038] A program according to an eleventh aspect of the present disclosure is a program for causing a computer to execute the information processing method according to any one of the first to tenth aspects.
[0039] This makes it possible to achieve the same effect as the above-described information processing method using a computer.
[0040] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.
[0041] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims will be described as optional components. Note that each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.
[0042] In the following description, elements may be assigned ordinal numbers such as first, second, and third. These ordinal numbers are assigned to elements to distinguish them from one another and do not necessarily correspond to a meaningful order. These ordinal numbers may be rearranged, newly added, or removed as appropriate.
[0043] (Embodiment 1) [Overview] First, an overview of a sound registration system according to an embodiment will be described. FIG. 1 is a schematic diagram illustrating a use example of the sound registration system according to embodiment 1. FIG. 1 illustrates a user performing an operation to register a sound in the sound registration system. In the following description, an example will be described in which a video, which is an example of a time change in the sound generation situation, is used as sound-related information related to input to and output from the information processing device 110. However, sound-related information other than a video may be used for the input to and output from the information processing device 110. For example, in addition to a video, sound-related information may be a spectrogram (in which frequency characteristics are represented as a time change in the characteristics of the sound itself, with frequency on the vertical axis, time on the horizontal axis, and intensity as shades of gray, etc.), which is a time change in the characteristics of the sound itself, or power (sound intensity). If the target sound is loud, the waveform of the sound wave itself may be used. The sound-related information may be any information that can simultaneously indicate at least information (such as the situation in which the sound is generated, frequency characteristics, power, or waveform) that allows for estimation of the sound being generated at each time point for two or more time points. Such sound-related information allows the user to identify the time point at which the desired sound is thought to be generated from among two or more time points. In other words, the information used for the sound-related information is required to enable a bird's-eye view (across multiple time points) of the time domain at which the desired sound is generated. Furthermore, the sound-related information is preferably information that can be recognized visually so that two or more time points can be recognized simultaneously. Note that, when sound-related information other than video is used for input to the information processing device 110, for example, a conversion process from audio or video may be performed on an application of the terminal device 300 (see FIG. 2 described later), and the sound-related information may be generated and transmitted to the information processing device 110.
[0044] As shown in FIG. 1 , in the sound registration system according to the present embodiment, a user uses a terminal device 300 to capture video of a machine operating. The captured video is then transmitted to an information processing device 110 (see FIG. 2 , described later), where it is processed to extract sounds associated with specific actions from the video (separate them from other sounds). The resulting separated sounds are registered in a sound discrimination device 120 (see FIG. 2 , described later). The sound discrimination device 120 can use the registered sounds as reference sounds and compare them to determine whether the sounds are identical to the reference sounds. The sounds registered in the sound discrimination device 120 are determined by the user based on the display information displayed after processing by the information processing device 110. In the display information, sounds are clustered into two or more clusters based on whether they are similar sounds. The user simply needs to determine whether each cluster is the desired sound. However, sounds included in the same cluster may include sounds that are not consecutive in time series. Therefore, in the sound registration system described below, sounds are displayed in a distinguishable manner for each cluster, and are arranged in a time series, allowing the user to determine the range in which the target sound is playing (i.e., the range to be registered) on that time series. In this way, the user can determine and register an appropriate sound as the target sound while viewing the displayed information, making it possible for the sound registration system to easily and appropriately obtain sounds for registration.
[0045] [Configuration] A specific configuration of the sound registration system will be described below with reference to Fig. 2 and subsequent figures. Fig. 2 is a block diagram showing the functional configuration of the sound registration system according to embodiment 1. As shown in Fig. 2, the sound registration system is realized as a server device 100. The server device 100 is a computer configured to be able to communicate with the terminal device 300, and is realized as an edge server or a cloud server.
[0046] The terminal device 300 is another information processing device used for inputting and outputting information between the information processing device 110 in the server device 100 and a user. Examples of the terminal device 300 include a smartphone, a tablet device, and a PC. The terminal device 300 includes an imaging unit 301, a display unit 302, and a sound collection unit 303. While the terminal device 300 will be described as a single device including all of the imaging unit 301, the display unit 302, and the sound collection unit 303, the terminal device 300 may be configured as a device group including three or more devices, such as a device including only the imaging unit 301 among the imaging unit 301, the display unit 302, and the sound collection unit 303, a device including only the display unit 302 among the imaging unit 301, the display unit 302, and the sound collection unit 303, and a device including only the sound collection unit 303 among the imaging unit 301, the display unit 302, and the sound collection unit 303.
[0047] The image capturing unit 301 is composed of an image sensor that captures images and a communication interface for communicating with the outside. The image capturing unit 301 captures images continuously in time series, and is linked to the sound collecting unit 303, so that it can generate a video that includes sounds collected by the sound collecting unit 303.
[0048] The display unit 302 is configured with a display for displaying images, a processing circuit for generating images to be displayed on the display, and a communication interface for communicating with the outside. The display unit 302 is used to present information to the user for inputting and outputting information between the information processing device 110 and the user, and displays display information to be presented to the user (an example of information including sound-related information and cluster information) as an image.
[0049] The sound collection unit 303 is configured with a microphone that collects sound and a communication interface for communicating with the outside. The sound collection unit 303 is linked to the image capture unit 301, and can generate a video including consecutive images in chronological order that are captured by the image capture unit 301.
[0050] The terminal device 300 can communicate with the server device 100 via a network 150. The network 150 here may be a wide-area communication network such as the Internet, a local communication network such as an intranet within a facility, or infrared or 2.4 GHz band short-range wireless communication. There are no particular limitations on the communication method used in the network 150. The image capturing unit 301, the display unit 302, and the sound collecting unit 303 can each communicate with the server device 100 via a communication interface and the network 150 to send and receive information.
[0051] The server device 100 includes an information processing device 110 and a sound discrimination device 120. A part of the configuration of the server device 100 can be said to implement a sound registration system.
[0052] The information processing device 110 is realized by executing a predetermined program using a processor and a memory. The information processing device 110 includes, as functional parts that are realized by executing the program, an acquisition unit 111, a calculation unit 112, an output unit 113, a detection unit 114, a clustering unit 115, and a registration unit 116.
[0053] The acquisition unit 111 is a functional part that acquires the video generated by the terminal device 300. The acquisition unit 111 acquires the video and separates the video into images and sounds. In other words, the acquisition unit 111 extracts sounds from the acquired video. The extracted sounds are further divided into unit sounds of unit time that are used in subsequent processing. Note that the unit time may be set appropriately depending on the subsequent processing; for example, in this example, the sound is divided into unit sounds of 2 seconds each. Even when divided into unit sounds in this way, each unit sound is linked so that it is possible to identify which part of the original video each unit sound corresponds to.
[0054] In this example, when dividing the sound into unit sounds, the sound is divided every two seconds, but two consecutive unit sounds may or may not have at least a partial overlap. In the former case, of two consecutive unit sounds, the leading first unit sound is shifted, for example, by several hundred milliseconds from the following second unit sound, resulting in an overlap of one second or more. Therefore, the first and second unit sounds have a sound difference of several hundred milliseconds before and after, in that the second unit sound does not have the leading several hundred milliseconds of the first unit sound, and the first unit sound does not have the trailing several hundred milliseconds of the second unit sound, but they share a sound of one second or more in between. In the latter case, of two consecutive unit sounds, the leading first unit sound is composed only of sounds that are completely different in time from the following second unit sound.
[0055] The calculation unit 112 is a functional part that calculates feature quantities of sounds included in a video as vectors (feature quantity vectors). When calculating feature quantity vectors from sounds included in a video, the calculation unit 112 calculates a vector having feature quantities with a first dimension (first-dimensional vector). In this embodiment, the first-dimensional vector is used as the feature quantity vector as is. On the other hand, in a second embodiment described below, the first-dimensional vector is created by a first-dimensional data creation unit 1121 (see FIG. 5 described below), and this is converted by a further functional part to calculate and use a vector with a different number of dimensions from the first-dimensional vector as the feature quantity vector. Therefore, in this embodiment, the first-dimensional vector and the feature quantity vector are treated as the same thing, whereas in the second embodiment described below, the first-dimensional vector and the feature quantity vector are treated as different things. From the above, it can be said that in this embodiment, the calculation unit 112 operates in a first mode in which a feature vector is calculated using only the first-dimensional data creation unit 1121 of the calculation unit 112, and in embodiment 2, the calculation unit 112 operates in a second mode in which a feature vector is calculated using the first-dimensional data creation unit 1121 and an additional functional part of the calculation unit 112. Therefore, if the calculation unit 112 includes the first-dimensional data creation unit 1121 and an additional functional part, the calculation unit can operate by switching between the first mode and the second mode.
[0056] The first-dimensional data creation unit 1121 of the calculation unit 112 has a trained model, such as OpenL3, that receives sound as input and outputs a first-dimensional vector corresponding to the sound, which is made up of 512-dimensional (i.e., the first dimension is 512) feature quantities. The first-dimensional data creation unit 1121 inputs unit sounds into this trained model and calculates a first-dimensional vector by outputting a first-dimensional vector for each unit sound. The above is just one example, and the first-dimensional data creation unit 1121 may be configured in any way as long as it can calculate a vector having first-dimensional feature quantities corresponding to processing from sound.
[0057] The output unit 113 is a functional part that creates and outputs display information for displaying to the user, including moving images and cluster information. Specific examples of the display information will be described later.
[0058] The detection unit 114 is a functional part that detects the selection of a registration target range by the user when the display information is displayed to the user. In other words, the detection unit 114 receives an input from the user for selecting a registration target range after the display information is displayed to the user.
[0059] The clustering unit 115 is a functional part that clusters sounds into two or more clusters based on feature vectors. For example, the clustering unit 115 determines, for each unit sound, which cluster out of a predetermined number of clusters the unit sound belongs to, using an algorithm called K-means, and assigns information to the unit sound indicating which cluster it belongs to (i.e., cluster information).
[0060] The registration unit 116 is a functional part that registers sounds contained in a video that fall within the registration target range as registration sounds and outputs them separately from sounds that do not fall within the registration target range in the sound discrimination device 120.
[0061] The sound discrimination device 120 is a device that discriminates whether a sound is similar to the reference sound or different from the reference sound by comparing it with the reference sound, and is realized by executing a predetermined program using a processor and memory. The sound discrimination device 120 also includes a storage unit 121 realized by a semiconductor memory or the like. The storage unit 121 stores the sound output by the registration unit 116. In this way, the sound is registered.
[0062] The registration unit 116 may output the sound in a format corresponding to the form of sound discrimination in the sound discrimination device 120. The registration unit 116 may output the sound as is and register it in the storage unit 121 as described above, or may, for example, acquire from the first-dimensional data creation unit 1121 a vector having first-dimensional features corresponding to sounds falling within the registration target range, output this as a sound to be registered, and register it in the storage unit 121. Alternatively, the registration unit 116 may acquire from the feature creation unit 1122 a feature vector having second-dimensional features corresponding to sounds falling within the registration target range, output this as a sound to be registered, and register it in the storage unit 121. In this way, sound registration is a broad concept that includes not only the registration of the sound itself, but also the registration of information about the sound, such as the analysis results of the sound.
[0063] The sound discrimination device 120 includes a functional part for discriminating sounds (not shown), which is implemented by a processor and a memory executing a predetermined program. Depending on the configuration of this functional part for discriminating sounds, a method for using registered sounds as reference sounds is appropriately set. For example, the sound discrimination device 120 is configured such that the first-dimensional vectors of each registered sound are linked and registered. The sound discrimination device 120 calculates the first-dimensional vector of a target sound to be discriminated, and compares the calculation result with the registered first-dimensional vector to determine whether the target sound is similar to the reference sound or a different sound from the reference sound. Alternatively, the sound discrimination device 120 includes a clustering model trained using the registered sounds. The sound discrimination device 120 inputs the target sound into the clustering model, which outputs whether the target sound is clustered in the same cluster as the reference sound, and can determine whether the target sound is similar to the reference sound or a different sound from the reference sound based on the output result.
[0064] [Operation] Details of the sound registration operation in this embodiment will be described below with reference to Figures 3 and 4 in addition to Figure 2. Figure 3 is a flowchart showing an example of the operation of the sound registration system according to embodiment 1. In Figure 3, the information processing device 110, other than the calculation unit 112 and the clustering unit 115, i.e., the acquisition unit 111, output unit 113, detection unit 114, and registration unit 116, are collectively shown as input / output units. Also, in Figure 3, arrows (arrows extending in the left-right direction on the page) between components indicate that information selected, generated, or output in the immediately preceding step is being transmitted to the indicated component.
[0065] As shown in FIG. 3 , a user uses a terminal device 300, more specifically, a camera unit 301 and a sound collection unit 303, to capture a video of a machine operating, including the sound of the machine, performing an operation of which the reference sound is to be obtained. For example, the user may capture several videos, select a video that was successfully captured (step S101), and transmit the video to the information processing device 110. In the information processing device 110, an acquisition unit 111 acquires the video, extracts audio from the video, and divides the audio into unit sounds (step S102). The division into unit sounds may involve dividing the video into unit times and then extracting only the audio, or may involve extracting audio from the video and then dividing the audio into unit times. The acquisition unit 111 transmits the unit sounds to a calculation unit 112. In the calculation unit 112, a first-dimensional data creation unit 1121 calculates a vector having a first-dimensional feature (first-dimensional vector) (step S103).
[0066] The first-dimensional data creation unit 1121 transmits the calculated first-dimensional vector to the clustering unit 115. The clustering unit 115 performs clustering of the unit sounds and creates cluster information for each unit sound (step S104). The clustering unit 115 transmits the cluster information to the output unit 113. The output unit 113 creates display information including the acquired video and the created cluster information (step S105). The output unit 113 outputs the created display information and transmits it to the terminal device 300. The display unit 302 displays the received display information (step S106). Here, Figure 4 is a diagram showing an example of a screen displayed on a terminal device in the sound registration system according to embodiment 1. Figure 4 shows the terminal device 300 on which display information is displayed.
[0067] As shown in Figure 4, the display information displayed on the display of the display unit 302 includes a video timeline in which video thumbnails (multiple images that make up the video) are arranged in chronological order, a preview of the video at the current position that can be set to any position on this video timeline, cluster information, and a registration button.
[0068] In addition to the current position, the video timeline allows you to set a registration target range (target range). Anything outside the target range is automatically set as a non-target range. As the target range, if you determine the start and end points on the video timeline, the section between the start and end points is automatically set. Multiple sets of start and end points may also be set. In this case, the section between the closest combination of start and end points (only in this order) is automatically set as the target range. Note that the user can play the video to set the target range, and the video being played can be checked in the video preview. Furthermore, while the video is playing, the current position automatically moves to the position corresponding to the frame in the video being played.
[0069] Here, the cluster information is displayed as a marker attached to a corresponding position in the time series on the video timeline. The cluster information can be displayed not only directly below the video timeline but also at any position, such as directly above or within the video timeline, as long as the chronological correspondence between the cluster information and the video timeline can be grasped. Each marker is displayed in a color (here, indicated by hatching) that indicates whether the unit sound belongs to the first cluster or the second cluster. That is, the display color of the marker differs depending on which cluster the marker belongs to. While viewing this screen (display information), the user can identify the position on the video timeline that corresponds to the target sound and determine the start and end points. In this case, if a specific action is observed in the video, the user can first set the location where the action is being performed as the registration target range. Then, the user can determine the start and end points on the video timeline so that the start point is the position where a sound with cluster information identical to the cluster information of the location where the action is being performed starts, and the end point is the position where a sound with cluster information identical to the cluster information of the location where the action is being performed ends. In this way, it is possible to easily determine the range to be registered by guessing which cluster in the video corresponds to the target sound and determining the start and end points using the cluster information.
[0070] Returning to FIG. 3 , the user sets a registration target range according to the display information and confirms the registration target range by pressing the registration button, i.e., selecting a portion of the video and confirming the selection (step S107). The terminal device 300 is equipped with input devices such as a touch panel and a mouse for accepting selections. Information indicating the selection of a portion of the video as the selected registration target range and the pressing of the registration button are transmitted to the information processing device 110 and detected by the detection unit 114. The registration unit 116 separates sounds included in the video that fall within the confirmed registration target range as sounds for registration from sounds that do not fall within the registration target range (step S108). The registration unit 116 outputs the separated sounds and transmits them to the sound discrimination device 120. The sound discrimination device 120 registers the sounds by storing the transmitted sounds in the memory unit 121 (step S109).
[0071] (Embodiment 2) The following describes embodiment 2, focusing on the differences from embodiment 1. Note that in the following description, the contents already described in embodiment 1, i.e., the configuration that is substantially the same as embodiment 1, may be omitted or simplified.
[0072] [Configuration] The sound registration system according to the present embodiment is similar to the above-described embodiments except for the configuration of the calculation unit 112 of the information processing device 110. Fig. 5 is a block diagram showing the functional configuration of the calculation unit of the information processing device according to the second embodiment.
[0073] 5, the calculation unit 112a in this embodiment includes a feature quantity creation unit 1122 and a two-dimensional data creation unit 1123 in addition to a first-dimensional data creation unit 1121. The first-dimensional data creation unit 1121 has already been described, so its description will be omitted here.
[0074] A first-dimensional vector, such as 512 dimensions, created by the first-dimensional data creation unit 1121 cannot be drawn in a diagram, making it difficult for a user to visually understand the relationship between vectors, for example, using a graph. In other words, if the vector information is converted into user-understandable information, it can be displayed and used by the user to determine the registration target range. In this embodiment, the feature creation unit 1122 performs vector transformation to calculate a feature vector having features with a second dimension, which is a lower dimension (lower order) than the first dimension, such as 512 dimensions. For this vector transformation, for example, t-SNE (t-Distributed Stochastic Neighbor Embedding) can be used. As a result, the second dimension of the feature vector can be set to 2 or 3 while maintaining the similarity between vectors as the distance between the vectors as much as possible, allowing the feature vector to be graphed. In other words, the user can visually understand the feature vector as a graph. Note that any transformation may be performed for the vector transformation here as long as it can reduce the number of dimensions below the first dimension. In the following description, it is assumed that the first-dimensional vector is converted into a two-dimensional vector by the feature creation unit 1122 .
[0075] Incidentally, a first-dimensional vector, such as 512 dimensions, created by the first-dimensional data creation unit 1121 can be directly subjected to clustering, as described below. However, if the first-dimensional vector is directly subjected to clustering, the amount of processing required for the clustering calculations becomes enormous. As described above, if the feature creation unit 1122 converts the first-dimensional vector into a two-dimensional vector, it is possible to solve the problem of the difficulty for users to visually understand the relationship between vectors using graphs, etc., as well as the problem of the enormous amount of processing required when directly subjecting the first-dimensional vector to clustering. On the other hand, to solve the problem of the enormous amount of processing required when directly subjecting the first-dimensional vector to clustering, it may not be necessary to convert it into a two-dimensional vector.
[0076] For example, if a feature vector has features with fewer dimensions than the first dimension (hereinafter referred to as "reduced dimensions" for convenience), clustering with accuracy (i.e., high accuracy) similar to that of the original first-dimensional vector is possible, while also reducing the amount of processing. For example, when calculating a feature vector, depending on the processing performance of the system used for the calculation, an appropriate number of dimensions can be selected from among a number of dimensions that is fewer than the first dimension and greater than 2 (e.g., 50 dimensions as the reduced number for a first dimension of 512), and processing can be performed to convert the feature vector into the reduced-dimensional feature vector. The reduced-dimensional feature vector can then be used to create cluster information, as described below. This makes it possible to appropriately adjust the balance between clustering accuracy and the amount of processing required for the calculation.
[0077] The calculation unit 112 uses the feature vectors converted into two-dimensional vectors to create relationship information indicating the relationships between the feature vectors corresponding to the unit sounds. Specifically, the two-dimensional data creation unit 1123 creates two-dimensional data as the relationship information.
[0078] In the two-dimensional data created by the two-dimensional data creator 1123, the similarity between the original first-dimension number vectors is represented in a two-dimensional graph as the proximity of the distance between the vectors while minimizing the loss of the similarity between the original first-dimension number vectors. In other words, the closer the distance between the vectors in the two-dimensional data, the higher the similarity between the first-dimension number vectors.
[0079] Furthermore, in this embodiment, the clustering unit 115 performs clustering not based on the first-dimensional vector but on the feature vector obtained by conversion from the first-dimensional vector. Therefore, compared to embodiment 1, the clustering may be less accurate due to the amount of information lost due to the vector conversion. However, since the cluster information obtained as a result of this clustering corresponds to the two-dimensional data, the registration target range can be determined by also referring to the two-dimensional data. This may result in a better selection of the actual target sound than if the registration target range were determined based solely on the cluster information.
[0080] [Operation] In the operation of the sound registration system according to the present embodiment, steps S101 to S103 and the operations following step S106 are the same as those in the first embodiment. However, steps S103 to S105 differ from those in the first embodiment. FIG. 6 is a flowchart showing an example of the operation of the sound registration system according to the second embodiment. As shown in FIG. 6, in the present embodiment, after step S103, the first-dimensional data creation unit 1121 transmits the calculated first-dimensional vector to the feature creation unit 1122. The feature creation unit 1122 performs vector transformation to reduce the number of dimensions of the first-dimensional vector, thereby calculating a feature vector (step S201). The feature creation unit 1122 transmits the calculated feature vector to the two-dimensional data creation unit 1123 and the clustering unit 115. The clustering unit 115 creates cluster information (step S202). Furthermore, the two-dimensional data creation unit 1123 creates two-dimensional data (step S203). Then, the clustering unit 115 and the two-dimensional data creation unit 1123 transmit the created cluster information and the created two-dimensional data to the output unit 113. The output unit 113 creates display information including the acquired video, the created cluster information, and the created two-dimensional data (step S204). The subsequent processing is the same as in embodiment 1, so a description thereof will be omitted.
[0081] As a result, the display information shown in Fig. 7 is displayed. Fig. 7 is a diagram showing an example of a screen displayed on a terminal device in the sound registration system according to embodiment 2. Fig. 7 shows the terminal device on which the display information is displayed.
[0082] As shown in Figure 7, the display information displayed on the display of the display unit 302 includes a video timeline in which video thumbnails are arranged in chronological order, a preview of the video at the current position on this timeline that can be set to any position, cluster information, two-dimensional data, and a registration button.
[0083] The two-dimensional data here corresponds to the cluster information and is displayed in the same display format as the markers of the cluster information. Furthermore, in the two-dimensional data, when a user provisionally sets a start point and an end point to set a registration target range, the feature vectors corresponding to the provisional start point and end point may be displayed in a format different from that of other feature vectors (i.e., non-registration target ranges). This allows the user to visually confirm which feature vectors are included in the two-dimensional data when correcting the start point and end point of the registration target range determined by the cluster information.
[0084] Alternatively, the user may select feature vectors to be included in a target range on the two-dimensional data, or both the setting of start and end points on the video timeline and the selection of feature vectors on the two-dimensional data may be possible. This makes it easier to select similar feature vectors on the two-dimensional data even if they are not temporally continuous. The selection of feature vectors on the two-dimensional data may be performed by setting a selection range on the two-dimensional data to select feature vectors included within the range, or by setting a non-selection range to prevent selection of feature vectors included within the range. The selection or non-selection range may be set by operating a mouse cursor or the like on the GUI to expand or contract a predetermined shape such as an ellipse or rectangle, or by setting a range of any shape using a so-called lasso tool. Alternatively, the feature vector displayed on the two-dimensional data may be toggled between selected and non-selected by tapping or clicking the feature vector.
[0085] Other Embodiments Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.
[0086] For example, in the above embodiment, the sounds of the feature vectors corresponding to the first cluster and the sounds of the feature vectors corresponding to the second cluster are colored, thereby enabling the sounds of the feature vectors corresponding to the first cluster and the sounds of the feature vectors corresponding to the second cluster to be displayed in a graph that is distinguishable from each other by different display colors. However, the sounds of the feature vectors corresponding to the first cluster and the sounds of the feature vectors corresponding to the second cluster may also be distinguished by different display shapes. FIG. 8 is a diagram partially illustrating an example of a screen displayed on a terminal device in a sound registration system according to another embodiment. FIG. 8 shows a video timeline according to another example under the same circumstances as FIG. 4 . As shown in FIG. 8 , the sounds of the feature vectors corresponding to the first cluster may be plotted as triangles and the sounds of the feature vectors corresponding to the second cluster may be plotted as circles, thereby enabling the sounds of the feature vectors corresponding to the first cluster and the sounds of the feature vectors corresponding to the second cluster to be displayed in a distinguishable manner by different display shapes. Furthermore, different display shapes and different display colors may be combined.
[0087] Furthermore, in addition to the first and second clusters, the clustering unit may cluster sounds into three or more clusters, such as a third cluster. FIG. 9 is a diagram partially illustrating an example of a screen displayed on a terminal device in a sound registration system according to another example of the embodiment. FIG. 9 illustrates a video timeline according to another example under the same circumstances as FIG. 4 . As illustrated in FIG. 9 , in the video timeline, sounds with feature vectors corresponding to the first cluster, sounds with feature vectors corresponding to the second cluster, and sounds with feature vectors corresponding to the third cluster are clustered. In other words, the clustering unit has identified the sounds included in the video as three types of sounds. Note that the clustering unit here does not necessarily use an algorithm that specifies the number of clusters, such as K-means, but may use an algorithm that automatically sets the number of clusters when classifying sounds. In the diagram, the user considers the third cluster and the first cluster to be the same sound, and the user selects the sounds that make up the first and third clusters as target sounds. In this way, if the clustering unit is configured to classify sounds into three or more clusters, it becomes possible to determine whether each sound in each cluster is a target sound and to register the sound appropriately.
[0088] Although the above description focuses on calculating a feature vector using mainly sound information from a video, this is not limiting. Since OpenL3, used to calculate the first-dimensional vector, is trained using both sound and image inputs, the first-dimensional vector may be calculated using both sound information and image information from the video. In this case, a first-dimensional vector of, for example, 512 dimensions may be obtained from the sound information, and a first-dimensional vector of, for example, 512 dimensions may be obtained from the image information. Similar operations can be performed by concatenating these first-dimensional vectors and subjecting the resulting 1024-dimensional concatenated first-dimensional vector to subsequent processing.
[0089] Furthermore, a model such as OpenL3 may not be used when calculating a first-dimensional vector from image information. For example, the image information may include brightness value information at each two-dimensional point, and this information may be used to calculate the first-dimensional vector. As an example, when a machine is photographed with a fixed camera to obtain a video, observing a specific pixel area in the image information included in the video reveals differences in brightness values indicating whether a material, product, or the like is placed in that pixel area as the machine operates. This clearly shows that brightness values are related to the operating state of the machine, so it is effective to use this information in combination with sound information as described above. Note that, depending on the type of machine, the first-dimensional vector may be calculated using only image information instead of sound information, and processing similar to that described in the above embodiment may be performed.
[0090] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.
[0091] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.
[0092] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.
[0093] Furthermore, the general or specific aspects of the present disclosure may be realized as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of an apparatus, a method, an integrated circuit, a computer program, and a recording medium.
[0094] For example, the present disclosure may be realized as an information processing method executed by a computer, or as a program for causing a computer to execute the information processing method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.
[0095] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.
[0096] The present disclosure is useful in reproducing sound, for example, by allowing a user to perceive stereoscopic sound.
[0097] REFERENCE SIGNS LIST 100 Server device 110 Information processing device 111 Acquisition unit 112, 112a Calculation unit 113 Output unit 114 Detection unit 115 Clustering unit 116 Registration unit 120 Sound discrimination device 121 Storage unit 150 Network 300 Terminal device 301 Photography unit 302 Display unit 303 Sound collection unit 1121 First-dimensional number data creation unit 1122 Feature amount creation unit 1123 Two-dimensional data creation unit
Claims
1. An information processing method executed by a computer for assisting in the registration of a sound as a reference sound in a sound discrimination device that discriminates sounds by learning reference sounds, comprising the steps of: acquiring sound-related information that includes a sound and represents at least one of a time change in the occurrence status of the sound or a time change in the characteristics of the sound; calculating a feature vector of the sound included in the sound-related information; clustering the sounds included in the sound-related information into two or more clusters based on the feature vector, and creating cluster information indicating to which of the two or more clusters the sounds belong; and creating and outputting information that includes the sound-related information and the cluster information.
2. The information processing method of claim 1, wherein the information includes: as the sound-related information, a timeline in which the sound occurrence status at each of multiple points in time that constitute the sound-related information or the characteristics of the sounds are arranged in chronological order; and as the cluster information, markers attached to corresponding positions in the chronological order on the timeline, the display manner of the markers differing depending on which of the two or more clusters the markers belong to.
3. The information processing method of claim 1, wherein in the step of calculating the feature vector, a first-dimensional vector having features of a first dimension is calculated from the sounds included in the sound-related information, and the feature vector having features of a number of dimensions less than the first dimension is calculated by converting the calculated first-dimensional vector, and in the step of creating cluster information, the sounds included in the sound-related information are clustered into two or more clusters based on the feature vector having features of a number of dimensions less than the first dimension.
4. The information processing method of claim 3, wherein the step of calculating the feature vector comprises calculating a first-dimensional vector having features of a first dimension from the sounds included in the sound-related information, and converting the calculated first-dimensional vector to calculate the feature vector having features with a dimension of two, which is less than the first dimension; the step of creating the cluster information further comprises a step of clustering the sounds included in the sound-related information into two or more clusters based on the feature vector having a feature with a dimension of two, and creating two-dimensional data in which similarities between each of the feature vectors are expressed as two-dimensional distances between each of the feature vectors; and the information further includes the created two-dimensional data.
5. The information processing method according to claim 1, wherein the feature vector is a plurality of vectors corresponding to unit sounds obtained by dividing the sounds included in the sound-related information into unit times.
6. The information processing method according to claim 1, wherein in the cluster information included in the information, the feature vectors belonging to one cluster among the two or more clusters can be distinguished from the feature vectors belonging to another cluster by difference in display color.
7. An information processing method according to any one of claims 1 to 6, wherein in the cluster information included in the information, the feature vectors belonging to one cluster among the two or more clusters can be distinguished from the feature vectors belonging to another cluster by differences in display shape.
8. An information processing method as described in claim 1, further comprising the steps of: detecting a selection of a portion of the time domain of the sound-related information as a registration target range in response to the display of the output information; and separating and outputting sounds included in the sound-related information that fall within the registration target range from sounds that do not fall within the registration target range, and registering the sounds that fall within the registration target range as sounds for registration.
9. An information processing device for assisting in the registration of a sound as a reference sound in a sound discrimination device that discriminates sounds by comparing them with a reference sound, comprising: an acquisition unit that acquires sound-related information including the sound and representing at least one of a time change in the occurrence status of the sound or a time change in the characteristics of the sound; a calculation unit that calculates a feature vector of the sound included in the sound-related information; a creation unit that clusters the sounds included in the sound-related information into two or more clusters based on the feature vector and creates cluster information indicating to which of the two or more clusters the sounds belong; and an output unit that creates and outputs information including the sound-related information and the cluster information.
10. A sound registration system comprising: a sound discrimination device that discriminates sounds by learning reference sounds; and an information processing device for assisting in the registration of sounds as the reference sounds, wherein the information processing device comprises: an acquisition unit that acquires sound-related information including sounds and representing at least one of a time change in the occurrence status of the sound or a time change in a feature of the sound; a calculation unit that calculates a feature vector of the sound included in the sound related information; a creation unit that clusters the sounds included in the sound related information into two or more clusters based on the feature vectors and creates cluster information indicating to which of the two or more clusters the sounds belong; an output unit that creates and outputs information including the sound related information and the cluster information; a detection unit that detects a selection of a part of a time domain of the sound related information as a registration target range in response to the output information being displayed; and a registration unit that registers sounds included in the sound related information that fall within the registration target range as sounds for registration, by separating them from sounds that do not fall within the registration target range and outputting them.
11. A program for causing a computer to execute the information processing method according to claim 1.
Citation Information
Patent Citations
Apparatus for signal classification
WO2011033597A1
Acoustic event recognition device, method, and program
WO2020054409A1
