Hammering sound determination device, model training device, hammering sound determination method, and program

Abstract

A hammering sound determination device according to an embodiment comprises a processor configured to execute: extracting, from time-series sound data obtained when inserting an implant into a bone, a plurality of pieces of partial data corresponding to hammering sounds; performing a calculation on each of the plurality of pieces of partial data to find, for a frequency band including a plurality of frequencies, a variable frequency band value at each frequency; generating input for a model on the basis of the values calculated for the frequency bands of the plurality of pieces of partial data; and determining the quality of the implant insertion on the basis of output obtained from the model.

Description

Tapping sound determination device, model learning device, tapping sound determination method, and program

[0001] An embodiment of the present invention relates to a tapping sound determination device, a model learning device, a tapping sound determination method, and a program.

[0002] Total hip arthroplasty (THA) is a surgical procedure that relieves pain and improves function in patients with osteoarthritis of the hip and femoral neck fractures. In THA, implants such as stems and rasps are inserted into the femur.

[0003] Improper implant insertion can lead to complications such as iatrogenic fractures and implant subsidence. However, because proper implant insertion requires certain techniques, there is a need for technology to assist in determining whether implant insertion is successful.

[0004] Patent Document 1 discloses a technology for assisting implant insertion, which predicts the amount of implant subsidence based on the sound pressure ratio of the low frequency band and the high frequency band relative to the sound pressure of the implant's tapping sound.

[0005] Japanese Patent Application Publication No. 2023-73662

[0006] Since the hammering sound of an implant is generated by a vibration phenomenon caused by physical contact between the hammer, the implant, and the human bone, if the physical characteristics of these elements change, the resonant frequency also changes. Therefore, with the above-mentioned technology, if the physical characteristics assumed in the prediction method change during the actual procedure, the accuracy of the pass / fail judgment may decrease.

[0007] An object of the embodiments of the present invention is to provide a technology for determining with high accuracy whether an implant has been inserted properly.

[0008] The tapping sound determination device according to the embodiment includes a processor configured to extract a plurality of partial data corresponding to a tapping sound from time-series sound data obtained when inserting an implant into a bone, calculate a value of a variable frequency band at each frequency for a frequency band including a plurality of frequencies for each of the plurality of partial data, generate an input to a model based on the calculated values ​​for the frequency bands of the plurality of partial data, and determine whether the implant is inserted properly based on an output obtained from the model.

[0009] According to the embodiment, it is possible to determine with high accuracy whether the insertion of the implant is successful or not.

[0010] FIG. 1 is a block diagram showing an example of the configuration of a beating sound detection system according to an embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of a beating sound detection device according to an embodiment. FIG. 3 is a block diagram showing an example of the hardware configuration of a model learning device according to an embodiment. FIG. 4 is a block diagram showing an example of the functional configuration of a beating sound detection device according to an embodiment. FIG. 5 is a diagram showing an example of the relationship between a feature map and a spectrogram generated by the beating sound detection device according to an embodiment. FIG. 6 is a diagram showing an example of a feature map generated by the beating sound detection device according to an embodiment. FIG. 7 is a diagram showing an example of a feature map generated by the beating sound detection device according to an embodiment. FIG. 8 is a block diagram showing an example of the functional configuration of a model learning device according to an embodiment. FIG. 9 is a flowchart showing an example of a beating sound detection process in the beating sound detection device according to an embodiment. FIG. 10 is a flowchart showing an example of a beating sound extraction process in the beating sound detection device according to an embodiment. FIG. 11 is a flowchart showing an example of a model learning process in the model learning device according to an embodiment. FIG. 12 is a diagram showing an example of a list of determination results using a teacher dataset by a determination model generated by the model learning device according to an embodiment. FIG. 13 is a diagram showing an example of a list of determination results using measurement data by a determination model generated by the model learning device according to an embodiment. Fig. 14 is a diagram showing an example of the relationship between a plurality of feature amount map candidates and feature amount maps generated by a beating sound determination device according to a modified example. Fig. 15 is a flowchart showing an example of a beating sound determination process in a beating sound determination device according to a modified example. Fig. 16 is a flowchart showing an example of a model learning process in a model learning device according to an embodiment. Fig. 17 is a diagram showing an example of a list of determination results using a teacher dataset according to a determination model generated by a model learning device according to a modified example. Fig. 18 is a diagram showing an example of a list of determination results using measurement data according to a determination model generated by a model learning device according to a modified example.

[0011] The embodiments will be described with reference to the drawings. Note that the scale of each part in the drawings used in the following description of the embodiments may be changed as appropriate. Also, for the sake of explanation, the drawings used in the following description of the embodiments may omit components.

[0012] 1. Embodiment 1.1 Configuration 1.1.1 Tapping Sound Determination System FIG. 1 is a block diagram showing an example of the configuration of a tapping sound determination system according to an embodiment.

[0013] The tapping sound determination system 1 is, for example, an operating room where total hip arthroplasty is performed. In the operating room, a surgeon uses a hammer H or the like to tap the implant I multiple times to insert the implant I into the femur B during total hip arthroplasty. Here, the implant I refers to any device inserted into a bone, such as a stem or rasp. The tapping sound determination system 1 also includes a sensor 2, a tapping sound determination device 3, and a model learning device 4.

[0014] The sensor 2 is, for example, a space propagation type or a contact type microphone. When a space propagation type microphone is applied, the sensor 2 is placed, for example, in an operating room, at a location several meters away from the source of the tapping sound. When a contact type microphone is applied, the sensor 2 is placed, for example, near the surgical site. Examples of the placement of a contact type microphone include (1) being fixed to a hammer, (2) being fixed with a band around the wrist of the surgeon, or (3) being fixed with a band around the patient's knee (in the case of femur surgery), etc. The sensor 2 measures sounds generated in the operating room and generates time-series sound data. The sensor 2 outputs the generated time-series data of the sound to the tapping sound determination device 3 as measurement data D.

[0015] The measurement data D includes various sounds generated in the operating room. Specifically, for example, the measurement data D includes, in addition to the tapping sound of the implant I, the sound generated when the implant I is pulled out (pulling sound), the surgeon's voice, noises emitted by equipment (not shown), and reverberations generated when these sounds are reflected on the walls of the operating room.

[0016] The tapping sound judgment device 3 is an information processing device such as a personal computer or a tablet. The tapping sound judgment device 3 extracts the tapping sound of the implant I from the measurement data D acquired by the sensor 2, and generates an input to the judgment model M based on the extracted tapping sound. The tapping sound judgment device 3 inputs the generated input to the judgment model M, and generates a judgment result R based on the obtained output. The tapping sound judgment device 3 presents the judgment result R to the surgeon to assist the surgeon in judging whether the insertion of the implant I into the femur B is successful.

[0017] The model learning device 4 is an information processing device such as a personal computer or a tablet. The model learning device 4 is configured to generate a determination model M. The model learning device 4 outputs the generated determination model M to the beating sound determination device 3.

[0018] The determination model M is a mathematical model including, for example, a neural network. The neural network in the determination model M includes a plurality of parameters. The plurality of parameters in the determination model M are optimized by the model learning device 4.

[0019] 1.1.2 Hardware Configuration Next, the hardware configuration of the beating sound determination system according to the embodiment will be described.

[0020] 2 is a block diagram showing an example of the hardware configuration of a tapping sound detection device 3 according to an embodiment. As shown in FIG. 2, the tapping sound detection device 3 includes a control circuit 11, a storage 12, a communication module 13, a user interface 14, a drive 15, and a storage medium 16.

[0021] The control circuit 11 is a circuit that controls the overall components of the beating sound detection device 3. The control circuit 11 includes a CPU (central processing unit), a RAM (random access memory), a ROM (read only memory), etc. The CPU of the control circuit 11 controls the overall beating sound detection device 3 in accordance with a program stored in the ROM of the control circuit 11. The RAM of the control circuit 11 has a working area for the CPU of the control circuit 11. The ROM of the control circuit 11 stores programs and the like used by the beating sound detection device 3.

[0022] The storage 12 includes, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage 12 stores information used in various processes in the beating sound determination device 3.

[0023] The communication module 13 is a circuit used for transmitting and receiving information between the tapping sound determination device 3 and the sensor 2 and the model learning device 4 .

[0024] The user interface 14 is a device that controls communication between the tapping sound detection device 3 and a user (e.g., a surgeon). The user interface 14 includes input devices and output devices. The input devices include, for example, a touch panel and operation buttons. The output devices include, for example, a printer, a speaker, and a display.

[0025] The drive 15 is a device for reading software stored in the storage medium 16. The drive 15 includes, for example, a CD (Compact Disk) drive or a DVD (Digital Versatile Disk) drive.

[0026] The storage medium 16 is a medium that stores software electrically, magnetically, optically, mechanically, or chemically. The storage medium 16 may store a program used by the beating sound determination device 3.

[0027] 3 is a block diagram showing an example of the hardware configuration of a model learning device 4 according to an embodiment. As shown in FIG. 3, the model learning device 4 includes a control circuit 21, a storage 22, a communication module 23, a user interface 24, a drive 25, and a storage medium 26.

[0028] The control circuit 21 is a circuit that provides overall control of each component of the model learning device 4. The control circuit 21 includes a CPU, RAM, ROM, etc. The CPU of the control circuit 21 controls the entire model learning device 4 in accordance with the programs stored in the ROM of the control circuit 21. The RAM of the control circuit 21 has a working area for the CPU of the control circuit 21. The ROM of the control circuit 21 stores programs and the like used by the model learning device 4.

[0029] The storage 22 includes, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage 22 stores information used in various processes in the model learning device 4.

[0030] The communication module 23 is a circuit used for transmitting and receiving information between the model learning device 4 and the tapping sound determination device 3 .

[0031] The user interface 24 is a device that manages communication between the model learning device 4 and the user. The user interface 24 includes input devices and output devices. The input devices include, for example, a touch panel and operation buttons. The output devices include, for example, a printer, a speaker, and a display.

[0032] The drive 25 is a device for reading software stored in the storage medium 26. The drive 25 includes, for example, a CD (Compact Disk) drive or a DVD (Digital Versatile Disk) drive.

[0033] The storage medium 26 is a medium that stores software electrically, magnetically, optically, mechanically, or chemically. The storage medium 26 may store a program used by the model learning device 4.

[0034] 1.1.3 Functional Configuration Next, the functional configuration of the tapping sound determination system according to the embodiment will be described.

[0035] <Tapping Sound Determination Device> Figure 4 is a block diagram showing an example of the functional configuration of a tapping sound determination device according to an embodiment. As shown in Figure 4, the CPU of the control circuit 11 loads a program stored in the ROM of the control circuit 11 or the storage medium 16 into the RAM of the control circuit 11. The CPU of the control circuit 11 then interprets and executes the program loaded into the RAM of the control circuit 11. As a result, the tapping sound determination device 3 functions as a computer including a reverberation sound removal unit 31, a spectrogram generation unit 32, a harmonic sound removal unit 33, a pluck sound removal unit 34, a feature map generation unit 35, and a determination unit 36.

[0036] The reverberation removal unit 31 is a functional block that removes reverberation in the operating room that is included in the measurement data D. The reverberation removal unit 31 removes reverberation from the measurement data D, for example, by applying an automorphic inverse filter to the measurement data D. Note that the reverberation removal unit 31 may perform preprocessing such as normalizing the sound pressure of the measurement data D or converting it to an appropriate frequency before removing the reverberation. The reverberation removal unit 31 sends the time-series sound data from which the reverberation has been removed to the spectrogram generation unit 32.

[0037] The spectrogram generation unit 32 is a functional block that generates a spectrogram. The spectrogram generation unit 32 generates a spectrogram, for example, by applying a fast Fourier transform to time-series sound data from which reverberation has been removed. A spectrogram is, for example, a group of three-dimensional data in which sound pressure is associated with a plurality of two-dimensional regions, each of which is associated with a pair of time and frequency. The spectrogram generation unit 32 sends the generated spectrogram to the harmonic sound removal unit 33.

[0038] The harmonic sound elimination unit 33 is a functional block that extracts percussion sounds by removing harmonic sounds from the sound data. The harmonic sound elimination unit 33 separates harmonic sound components and percussion components, for example, by applying harmonic percussive sound separation (HPSS) processing to the spectrogram. The harmonic sound elimination unit 33 extracts a spectrogram of the obtained percussion components. The obtained percussion data may contain not only the percussion sounds of the implant I, which are distributed in a relatively high frequency band, but also human speech components, which are distributed in a relatively low frequency band. Therefore, the harmonic sound elimination unit 33 may further remove a relatively low frequency band (e.g., a sound pressure spectrum of 800 Hz or less) from the obtained spectrogram of the percussion components. The harmonic sound elimination unit 33 sends the obtained spectrogram of the percussion components to the pluck sound elimination unit 34.

[0039] The plucked sound elimination unit 34 is a functional block that extracts tapping sounds by removing plucked sounds from the spectrogram of the percussion sound components. The sound pressure of plucked sounds is relatively low compared to that of percussion sounds. Therefore, the plucked sound elimination unit 34 separates tapping sounds from plucked sounds by, for example, applying Otsu's binarization method to the average sound pressure of the percussion sound components. The plucked sound elimination unit 34 then extracts multiple spectrograms (partial data), each corresponding to a tapping sound. The plucked sound elimination unit 34 sends the resulting multiple partial data to the feature map generation unit 35.

[0040] The feature map generation unit 35 is a functional block that generates features of the beating sound to be input to the determination model M. The feature map generation unit 35, for example, converts spectrograms corresponding to each of a plurality of partial data into a fluctuation frequency domain. The fluctuation frequency is a frequency representation of the time change of the peak value in a signal vibrating at a certain frequency. Specifically, the feature map generation unit 35 divides the spectrogram into a plurality of data groups corresponding to different frequency bands. The feature map generation unit 35 may then perform a Fourier transform process on each data group. The feature map generation unit 35 may also convert the spectrogram into the fluctuation frequency domain by using a plurality of band-pass filters that transmit different frequency bands, such as octave band analysis. Note that the fluctuation period of a beating sound is shorter than that of a continuously occurring sound. Therefore, while the minimum unit of analysis when performing a transformation process to the fluctuation frequency domain on a continuously occurring sound exceeds 100 milliseconds, it is preferable that the minimum unit of analysis when performing a transformation process to the fluctuation frequency domain be, for example, 100 milliseconds or less.

[0041] The feature map generator 35 performs a variation frequency conversion process on all data groups generated from one spectrogram to calculate multiple feature amounts, each of which is three-dimensional data on a frequency band, a variation frequency band, and sound pressure.

[0042] 5 is a diagram illustrating an example of the relationship between a spectrogram and a feature map generated by the beating sound determination device according to the embodiment, which illustrates the relationship between a spectrogram SPE corresponding to a certain frequency band fn and a portion of the feature map generated from the spectrogram SPE.

[0043] 5A, the spectrogram SPE corresponding to the frequency band fn forms an envelope ENV in the time domain. The feature map generator 35 converts the time axis of this envelope ENV into a fluctuating frequency axis, for example, using a Fourier transform. In other words, the fluctuating frequency is the frequency component when the envelope ENV is viewed as a composite of waves of different frequencies.

[0044] As shown in part (B) of Fig. 5, as a result of converting the time axis of the envelope ENV into the fluctuation frequency axis, amplitude (sound pressure or intensity) components for each fluctuation frequency band are obtained. In the example of Fig. 5, it is shown that the envelope ENV is obtained by combining a wave with an amplitude a1 in the fluctuation frequency band vf1, a wave with an amplitude a2 in the fluctuation frequency band vf2, a wave with an amplitude a3 in the fluctuation frequency band vf3, a wave with an amplitude a4 in the fluctuation frequency band vf4, and a wave with an amplitude a5 in the fluctuation frequency band vf5.

[0045] Then, as shown in part (C) of Fig. 5, the feature map generator 35 generates a feature map by mapping the obtained amplitude components a1 to a5 onto a two-dimensional plane of frequency F and fluctuation frequency VF. In the example of Fig. 5, the amplitude components a1 to a5 corresponding to the fluctuation frequency bands vf1 to vf5, respectively, are based on the spectrogram SPE corresponding to frequency band fn. Therefore, the amplitude components a1 to a5 are mapped in association with the regions where the fluctuation frequency bands vf1 to vf5 intersect with frequency band fn. In each region, the magnitude of the amplitude component can be represented, for example, by color intensity.

[0046] The feature map generating unit 35 generates a feature map by mapping the plurality of feature quantities to a plurality of two-dimensional regions, each of which corresponds to a set of a fluctuation frequency and a frequency. That is, the feature map generating unit 35 generates a plurality of feature maps, the number of which corresponds to the number of extracted beating sounds, from a plurality of spectrograms corresponding to the number of beating sounds. The feature map generating unit 35 sends the generated plurality of feature maps to the determining unit 36.

[0047] The determination unit 36 ​​is a functional block that determines whether the implant I has been inserted into the femur B properly based on multiple feature maps. The determination unit 36 ​​inputs one feature map into the determination model M and obtains one output. The output is, for example, a real value between 0 and 1, and indicates the probability that the implant I has been properly inserted into the femur B. In other words, the closer the output is to 1, the higher the probability that the procedure is appropriate, and the closer the output is to 0, the lower the probability that the procedure is appropriate. Note that an appropriate procedure means, for example, that the patient does not experience complications such as iatrogenic fractures or loosening of the artificial hip joint after surgery. Furthermore, an inappropriate procedure means, for example, that these complications occur.

[0048] The determination unit 36 ​​performs majority voting based on the multiple outputs for the multiple feature amount maps. That is, if the number of outputs equal to or greater than the threshold is greater than the number of outputs below the threshold, the determination unit 36 ​​outputs a determination result R indicating that the procedure is appropriate. On the other hand, if the number of outputs equal to or greater than the threshold is less than the number of outputs below the threshold, the determination unit outputs a determination result R indicating that the procedure is inappropriate. The threshold is a real number greater than 0 and less than 1. When the determination unit 36 ​​obtains the determination result R by majority voting, it is preferable that the number of feature amount maps input to the determination unit 36 ​​(i.e., the number of tapping sounds extracted from the measurement data D) is an odd number.

[0049] If the majority vote results in the determination that the procedure is appropriate, the determination result R output from the determination unit 36 ​​has a value of, for example, 1. If the majority vote results in the determination that the procedure is inappropriate, the determination result R output from the determination unit 36 ​​has a value of, for example, 0. The determination result R is not limited to the above example, and may be a calculation result (e.g., average value, weighted average, etc.) based on the output from the determination model M for the tapping sounds corresponding to the majority determination result in the majority vote process. More specifically, if the average value is applied as the calculation result, and the outputs for five tapping sounds are {0.4, 0.45, 0.8, 0.9, 0.94} and the threshold is 0.5, the determination result R may be calculated as 0.88 (= (0.8 + 0.9 + 0.94) / 3). The determination unit 36 ​​presents information indicating whether the procedure is appropriate to the surgeon based on the determination result R. In addition, the judgment result R may not only be presented to the surgeon, but may also be displayed on a display device outside the operating room via an in-hospital network not shown, or may be stored in storage 12 or an external storage device not shown.

[0050] 6 and 7 are diagrams showing examples of feature maps generated by the tapping sound determination device according to the embodiment. Fig. 6 shows an example of a feature map corresponding to a tapping sound for which an output close to 1 is obtained from the determination model M (i.e., a high probability that the technique is appropriate). Fig. 7 shows an example of a feature map corresponding to a tapping sound for which an output close to 0 is obtained from the determination model M (i.e., a low probability that the technique is appropriate).

[0051] In the example of Fig. 6, it can be seen that the sound pressure of the component of the tapping sound corresponding to the region Ga of the high frequency band and the low fluctuation frequency band is high. In the example of Fig. 7, it can be seen that the sound pressure of the component of the tapping sound corresponding to the region Gb of the low frequency band and the low fluctuation frequency band is high. As such, the features that appear in the feature map differ depending on whether the procedure is appropriate or not. The determination model M distinguishes the features that appear in the feature map for each tapping sound and determines whether the procedure is appropriate.

[0052] As described above, the determination model M includes a neural network. More specifically, the neural network included in the determination model M is, for example, a convolutional neural network (CNN). In this case, the feature map is input to the determination model M as image information such as a heat map. As a result, each feature constituting the feature map is treated by the determination model M as equivalent to a pixel value in an image.

[0053] <Model Learning Device> Fig. 8 is a block diagram showing an example of the functional configuration of a model learning device according to an embodiment. As shown in Fig. 8, the CPU of the control circuit 21 loads a program stored in the ROM of the control circuit 21 or the storage medium 26 into the RAM of the control circuit 21. The CPU of the control circuit 21 then interprets and executes the program loaded into the RAM of the control circuit 21. As a result, the model learning device 4 functions as a computer including a reverberation removal unit 41, a spectrogram generation unit 42, a harmonic sound removal unit 43, a plucked sound removal unit 44, a feature map generation unit 45, a determination unit 46, and an update unit 47. The model learning device 4 also stores a teacher dataset T and a pre-learning model M0.

[0054] The pre-training model M0 is a mathematical model including a neural network. The configuration of the pre-training model M0 is the same as that of the determination model M, except for the parameter values. The pre-training model M0 is a determination model in a state before the parameters are optimized for the tapping sound determination process.

[0055] The training data set T includes multiple training data. Each of the multiple training data includes a set of data and labels that are associated with each other. The data for each training data is, for example, data of multiple pre-recorded tapping sounds. Measurement data D can be used for the data for each training data. Therefore, multiple feature maps corresponding to the number of tapping sounds included in one training data can be obtained from the data. The label for each training data is information indicating the true judgment result for the corresponding data. Specifically, the label has a value of 0 or 1.

[0056] The configurations of the reverberation removal unit 41, spectrogram generation unit 42, harmonic sound removal unit 43, extraction sound removal unit 44, and feature map generation unit 45 are the same as the configurations of the reverberation removal unit 31, spectrogram generation unit 32, harmonic sound removal unit 33, extraction sound removal unit 34, and feature map generation unit 35, except that the input is teacher data.

[0057] The determination unit 46 inputs multiple feature maps obtained from one piece of training data into the pre-training model M0 and obtains multiple outputs. Then, similar to the determination unit 36, the determination unit 46 performs majority voting based on the multiple outputs. The determination unit 46 sends the result of the majority voting to the update unit 47.

[0058] The update unit 47 is a functional block that updates the parameters of the pre-training model M0 based on the result of the majority voting process performed by the determination unit 46. Specifically, the update unit 47 calculates an evaluation value based on the result of the majority voting process and the labels of the training data. The evaluation value is calculated, for example, using an evaluation function that is minimized when the result of the majority voting process matches the label. If the calculated evaluation function is equal to or greater than a threshold, the update unit 47 updates the parameters of the pre-training model M0. When updating the parameters, the update unit 47 uses, for example, an error backpropagation algorithm. The pre-training model M0, whose parameters have been updated by the update unit 47, is fed back to the determination unit 46. If the evaluation function becomes equal to or less than the threshold, the update unit 47 outputs the pre-training model M0 to the beating sound determination device 3 as the determination model M.

[0059] 1.2 Operation 1.2.1 Tap Sound Determination Process FIG. 9 is a flowchart showing an example of a tap sound determination process in the tap sound determination device according to the embodiment.

[0060] Upon receiving the measurement data D (start), the reverberation sound elimination unit 31, the spectrogram generation unit 32, the harmonic sound elimination unit 33, and the pluck sound elimination unit 34 execute a beating sound extraction process (S1). As a result, a plurality of partial data corresponding to a plurality of beating sounds are extracted from the measurement data D. The beating sound extraction process will be described in detail later.

[0061] The feature map generating unit 35 selects one tapping sound from the plurality of tapping sounds extracted in the process of S1 (S2).

[0062] The feature amount map generating unit 35 performs a variation frequency conversion process on the spectrogram of the beating sound selected in the process of S2, and generates a feature amount map (S3).

[0063] The determination unit 36 ​​inputs the feature map generated in the process of S3 into the determination model M and obtains an output (S4).

[0064] The determination unit 36 ​​determines whether the output obtained in the process of S4 is equal to or greater than a threshold value (S5).

[0065] If the output is equal to or greater than the threshold value (S5; yes), the determination unit 36 ​​increments the OK number (S6). Note that the initial value of the OK number is assumed to be 0.

[0066] If the output is less than the threshold value (S5; no), the determining unit 36 ​​increments the NG number (S7). Note that the initial value of the NG number is set to 0.

[0067] After the process of S6 or the process of S7, the determination unit 36 ​​determines whether or not all of the multiple beating sounds extracted in the process of S1 have been selected (S8).

[0068] If there are any unselected tapping sounds (S8; no), the feature map generating unit 35 selects an unselected tapping sound from the multiple tapping sounds extracted in the process of S1 (S2). Then, the subsequent processes of S3 to S8 are executed. In this way, the processes of S2 to S8 are repeatedly executed until all the tapping sounds have been selected.

[0069] If all the tapping sounds have been selected (S8; yes), the determining unit 36 ​​determines whether the number of OKs is greater than the number of NGs (S9).

[0070] If the number of OKs is greater than the number of NGs (S9; yes), the determination unit 36 ​​outputs information indicating that the procedure is appropriate as a determination result R (S10).

[0071] If the number of OKs is smaller than the number of NGs (S9; no), the determination unit 36 ​​outputs information indicating that the procedure is inappropriate as the determination result R (S11).

[0072] After the process of S10 or the process of S11, the beating sound determination process ends (END).

[0073] 1.2.2 Tap Sound Extraction Process Fig. 10 is a flowchart showing an example of the tap sound extraction process in the tap sound determination device according to the embodiment. The processes in S21 to S24 shown in Fig. 10 correspond to the process in S1 in Fig. 9.

[0074] When the tapping sound extraction process starts (START), the echo remover 31 applies an automorphic inverse filter to the measurement data D to remove the echo component from the measurement data D (S21).

[0075] The spectrogram generating unit 32 generates a spectrogram by applying a Fourier transform process to the sound data from which the reverberation components have been removed in the process of S21 (S22).

[0076] The harmonic sound removal unit 33 removes harmonic sound components by applying harmonic percussion sound separation processing to the spectrogram generated in the processing of S22 (S23).

[0077] The withdrawal sound removal unit 34 separates low sound pressure parts and high sound pressure parts by applying Otsu's binarization method to the spectrogram from which the harmonic sound components have been removed in the process of S23, and removes the low sound pressure parts as withdrawal sound components (S24).

[0078] After the process of S24, the plucked sound removal unit 34 sends the obtained spectrogram to the feature map generation unit 35, and the beating sound extraction process ends (END).

[0079] 1.2.3 Model Learning Process FIG. 11 is a flowchart showing an example of the model learning process in the model learning device according to the embodiment.

[0080] When the model learning process starts (START), the model learning device 4 selects training data from the training data set T (S31).

[0081] The reverberation sound elimination unit 41, spectrogram generation unit 42, harmonic sound elimination unit 43, and withdrawal sound elimination unit 44 execute a beating sound extraction process on the data in the training data selected in the process of S31 (S32). As a result, multiple pieces of partial data corresponding to multiple beating sounds are extracted from the training data. The beating sound extraction process of S32 is equivalent to the processes of S21 to S24 described using FIG. 10.

[0082] The feature map generating unit 45 selects one tapping sound from the plurality of tapping sounds extracted in the process of S32 (S33).

[0083] The feature amount map generating unit 45 performs a variation frequency conversion process on the spectrogram of the beating sound selected in the process of S33, and generates a feature amount map (S34).

[0084] The determination unit 46 inputs the feature map generated in the process of S34 into the pre-learning model M0 and obtains an output (S35).

[0085] The determination unit 46 determines whether the output obtained in the process of S35 is equal to or greater than a threshold value (S36).

[0086] If the output is equal to or greater than the threshold value (S36; yes), the determination unit 46 increments the OK number (S37). Note that the initial value of the OK number is set to 0.

[0087] If the output is less than the threshold value (S36; no), the determining unit 46 increments the NG number (S38). Note that the initial value of the NG number is set to 0.

[0088] After the process of S37 or the process of S38, the determination unit 46 determines whether or not all of the multiple beating sounds extracted in the process of S32 have been selected (S39).

[0089] If there are any unselected tapping sounds (S39; no), the feature map generating unit 45 selects the unselected tapping sounds from the plurality of tapping sounds extracted in the process of S32 (S33). Then, the subsequent processes of S34 to S39 are executed. In this way, the processes of S33 to S39 are repeatedly executed until all the tapping sounds have been selected.

[0090] If all the tapping sounds have been selected (S39; yes), the determination unit 46 determines whether the probability value obtained as a result of majority voting based on the number of OKs and the number of NGs matches the label (S40).

[0091] If the majority vote result is different from the label (S40; no), the update unit 47 updates the parameters of the pre-learning model M0 (S41).

[0092] If the majority vote result matches the label (S40; yes), or after the process of S41, the determination unit 46 determines whether the evaluation function at the time of parameter update satisfies a condition (S42). The condition may be, for example, that the evaluation function is equal to or less than a threshold value.

[0093] If the evaluation function does not satisfy the condition (S42; no), the model learning device 4 selects training data from the training data set T so as not to cause bias in the training data to be selected (S31). Then, the subsequent processes of S32 to S42 are executed. In this way, the processes of S31 to S42 are repeatedly executed until the evaluation function satisfies the condition.

[0094] If the evaluation function satisfies the condition (S42; yes), the update unit 47 outputs the pre-learning model M0 as the determination model M to the beating sound determination device 3 (S43).

[0095] After the process of S43, the model learning process ends (END).

[0096] 1.3 Effects of the Embodiment According to the embodiment, the reverberation sound removal unit 31 removes reverberation sound data from within the operating room from the measurement data D. The harmonic sound removal unit 33 separates the measurement data D, from which the reverberation sound data has been removed, into harmonic sound data and a plurality of hammering sound data, and removes the harmonic sound data. The pulling-out sound removal unit 34 removes pulling-out sound data when the implant I is pulled out from the plurality of hammering sound data. This allows the tapping sound determination device 3 to extract, in real time, a plurality of partial data pieces, each of which corresponds to a tapping sound when the implant I is inserted into the femur B, from the measurement data D acquired in the operating room.

[0097] Furthermore, the feature map generation unit 35 calculates, for each of the plurality of partial data, a plurality of feature amounts each associated with a pair of a frequency band and a variation frequency band. The feature map generation unit 35 generates a plurality of feature maps as input to the determination model M based on the plurality of feature amounts calculated for each of the plurality of partial data. The determination unit 36 ​​determines whether the insertion of the implant I is appropriate or not based on a plurality of outputs obtained by inputting the plurality of feature maps to the determination model M. This allows the tapping sound determination device 3 to present to the surgeon in real time whether the procedure is appropriate or not. Therefore, the surgeon can use the determination result R as support information for determining whether or not to continue adjusting the implant I.

[0098] Furthermore, the determination unit 36 ​​determines whether the insertion of the implant I is successful or not based on the result of majority voting of the multiple outputs. As a result, even if the extracted tapping sounds include sounds resulting from poor tapping, the determination unit 36 ​​can exclude such sounds and generate the determination result R. Therefore, it is possible to present the surgeon with a more reliable determination result R than when the determination result is calculated from a single tapping sound.

[0099] There are two possible cases where the judgment result R is incorrect: an overevaluation case where an appropriate procedure is judged to be inappropriate, and an oversight case where an inappropriate procedure is judged to be appropriate.

[0100] 12 is a diagram illustrating an example of an evaluation of a determination result using a teacher dataset by a determination model generated by a model learning device according to an embodiment. In the example of Fig. 12, the accuracy rate, the number of overestimates, and the number of overestimates when determining 42 teacher datasets T using a determination model M generated by applying majority voting processing are shown for each seed.

[0101] As shown in Figure 12, when the teacher dataset T was judged using the judgment model M generated by applying the majority voting process, the number of missed and over-evaluated cases was both 0, and the accuracy rate reached 100%.

[0102] 13 is a diagram illustrating an example of evaluation of a determination result using measurement data by a determination model generated by a model learning device according to an embodiment. In the example of FIG. 13, the accuracy rate, the number of overestimates, and the number of overestimates when 19 pieces of measurement data D are determined by a determination model M generated by applying majority voting processing are shown for each seed.

[0103] 13, when the measurement data D was judged using the judgment model M generated by applying the majority voting process, the number of overestimated cases was about four on average, but no overestimation occurred. The accuracy rate was about 80%.

[0104] As described above, when verification was performed using the configuration according to the embodiment, it was found that when the judgment result R is generated using the majority voting process, the overestimation case can be excluded with particularly high accuracy from the two cases described above. Therefore, according to the embodiment, it has been shown that by generating the judgment result R using the majority voting process, the overestimation case can be excluded with particularly high accuracy.

[0105] Furthermore, the feature map generating unit 35 sets the minimum analysis unit for converting the tapping sound into the fluctuation frequency domain to 100 milliseconds or less, which makes it possible to obtain a significant feature map even from a tapping sound whose fluctuation period is shorter than that of a sound that occurs continuously, such as the sound of a rotating motor.

[0106] Furthermore, the model learning device 4 can train the determination model M using various training data sets T that have different physical properties of the hammer H, implant I, and femur B. By using the determination model M, the tapping sound determination device 3 can thereby present the surgeon with a determination result R that is robust against changes in these physical properties.

[0107] 2. Modifications In the above embodiment, a case has been described in which majority voting processing is performed based on multiple outputs obtained by inputting multiple feature amount maps into the judgment model M, and the result of the majority voting processing is presented to the surgeon as the judgment result R, but this is not limited to this. A feature amount map to be input into the judgment model M may be generated based on multiple feature amount map candidates, and one output result obtained by inputting the generated feature amount map into the judgment model M may be presented to the surgeon as the judgment result R. The following mainly describes configurations and operations that differ from the embodiment. Descriptions of configurations and operations that are equivalent to those of the embodiment will be omitted as appropriate.

[0108] 2.1 Configuration <Tapping Sound Determination Device> The configurations of the reverberation sound removal unit 31, spectrogram generation unit 32, harmonic sound removal unit 33, and pluck sound removal unit 34 of the tapping sound determination device 3 according to the modified example are the same as those of the embodiment.

[0109] The feature amount map generating unit 35 generates a plurality of feature amount map candidates, each of which corresponds to a beat sound, from the plurality of spectrograms. The plurality of feature amount map candidates generated by the beat sound detection device 3 according to the modified example are equivalent to the plurality of feature amount maps generated by the beat sound detection device 3 according to the embodiment.

[0110] FIG. 14 is a diagram showing an example of the relationship between a plurality of feature amount map candidates and feature amount maps generated by the beating sound determination device according to the modified example.

[0111] 14 , the feature map generation unit 35 compares the sound pressures of the feature quantities corresponding to the same pair of frequency and fluctuation frequency among the multiple feature quantities constituting each feature map candidate, and selects the maximum value. The feature map generation unit 35 generates one feature map by combining the maximum values ​​of the feature quantities selected for each pair of frequency and fluctuation frequency. The feature map generation unit 35 sends the generated feature map to the determination unit 36.

[0112] The determination unit 36 ​​determines whether the insertion of the implant I into the femur B is appropriate based on one feature map. That is, when one output obtained by inputting the one feature map into the determination model M is equal to or greater than a threshold, the determination unit 36 ​​outputs a determination result R indicating that the procedure is appropriate. On the other hand, when one output obtained by inputting the one feature map into the determination model M is less than the threshold, the determination unit 36 ​​outputs a determination result R indicating that the procedure is inappropriate.

[0113] <Model Learning Device> The configurations of the reverberation sound removal unit 41, spectrogram generation unit 42, harmonic sound removal unit 43, and withdrawal sound removal unit 44 of the model learning device 4 according to the modified example, as well as the teacher data set T, are the same as those in the embodiment.

[0114] The feature map generator 45 generates a plurality of feature map candidates each corresponding to a beat sound from a plurality of spectrograms corresponding to a plurality of beat sounds included in one training data. The plurality of feature map candidates generated by the model learning device 4 according to the modified example are equivalent to the plurality of feature maps generated by the model learning device 4 according to the embodiment.

[0115] The feature map generation unit 45 compares the sound pressures of the feature quantities corresponding to the same pair of frequency and variation frequency among the multiple feature quantities that make up each feature map candidate, and selects the maximum value. The feature map generation unit 45 generates one feature map from one piece of training data by combining the maximum values ​​of the feature quantities selected for each pair of frequency and variation frequency. The feature map generation unit 35 sends the generated feature map to the determination unit 46.

[0116] The determination unit 46 inputs one feature map obtained from one piece of training data into the pre-training model M0, obtains one output, and sends the one output to the update unit 47 as an output result, similar to the determination unit 36.

[0117] The update unit 47 calculates an evaluation value based on the label of the training data and one output result obtained from the training data. The evaluation value is calculated, for example, using an evaluation function that is minimized when the output result and the label match. If the calculated evaluation function is equal to or greater than a threshold, the update unit 47 updates the parameters of the pre-training model M0. The pre-training model M0, whose parameters have been updated by the update unit 47, is fed back to the determination unit 46. If the training evaluation function becomes equal to or less than the threshold, the update unit 47 outputs the pre-training model M0 to the beating sound determination device 3 as the determination model M.

[0118] 2.2 Operation 2.2.1 Tap Sound Determination Process Fig. 15 is a flowchart showing an example of tap sound determination process in the tap sound determination device according to the modified example. Fig. 15 corresponds to Fig. 9 in the embodiment.

[0119] When the measurement data D is received (start), the reverberation sound remover 31, the spectrogram generator 32, the harmonic sound remover 33, and the plucked sound remover 34 execute the tapping sound extraction process (S51).

[0120] The feature amount map generating unit 35 selects one tapping sound from the plurality of tapping sounds extracted in the process of S51 (S52).

[0121] The feature amount map generating unit 35 performs a variation frequency conversion process on the spectrogram of the beating sound selected in the process of S52, and generates a feature amount map candidate (S53).

[0122] The determination unit 36 ​​determines whether or not all of the multiple beating sounds extracted in the process of S51 have been selected (S54).

[0123] If there are any unselected tapping sounds (S54; no), the feature map generating unit 35 selects an unselected tapping sound from the multiple tapping sounds extracted in the process of S51 (S52). Then, the subsequent processes of S53 and S54 are executed. In this way, the processes of S52 to S54 are repeatedly executed until all the tapping sounds have been selected.

[0124] If all of the tapping sounds have been selected (S54; yes), the feature map generating unit 35 generates a feature map based on the maximum value of the corresponding feature values ​​in the multiple feature map candidates generated in the process of S53 (S55).

[0125] The determination unit 36 ​​inputs the feature map generated in the process of S55 into the determination model M and obtains an output (S56).

[0126] The determination unit 36 ​​determines whether the output obtained in the process of S56 is equal to or greater than a threshold value (S57).

[0127] If the output is equal to or greater than the threshold value (S57; yes), the determination unit 36 ​​outputs information indicating that the procedure is appropriate as a determination result R (S58).

[0128] If the output is equal to or greater than the threshold value (S57; no), the determination unit 36 ​​outputs information indicating that the procedure is inappropriate as a determination result R (S59).

[0129] After the process of S58 or the process of S59, the beating sound determination process ends (END).

[0130] 2.2.2 Model Learning Process Fig. 16 is a flowchart showing an example of model learning process in the model learning device according to the modified example. Fig. 16 corresponds to Fig. 11 in the embodiment.

[0131] When the model learning process starts (START), the model learning device 4 selects training data from the training data set T (S61).

[0132] The reverberation sound elimination unit 41, spectrogram generation unit 42, harmonic sound elimination unit 43, and pluck sound elimination unit 44 execute a beating sound extraction process on the data in the training data selected in the process of S61 (S62), thereby extracting a plurality of partial data corresponding to a plurality of beating sounds from the training data.

[0133] The feature amount map generating unit 45 selects one tapping sound from the plurality of tapping sounds extracted in the process of S62 (S63).

[0134] The feature amount map generating unit 45 performs a variation frequency conversion process on the spectrogram of the beating sound selected in the process of S63, and generates a feature amount map candidate (S64).

[0135] The determination unit 46 determines whether or not all of the multiple beating sounds extracted in the process of S62 have been selected (S65).

[0136] If there are any unselected tapping sounds (S65; no), the feature map generating unit 45 selects the unselected tapping sounds from the plurality of tapping sounds extracted in the process of S62 (S63). Then, the subsequent processes of S64 and S65 are executed. In this way, the processes of S63 to S65 are repeatedly executed until all the tapping sounds have been selected.

[0137] If all of the tapping sounds have been selected (S65; yes), the feature map generating unit 45 generates a feature map based on the maximum value of the corresponding feature values ​​in the multiple feature map candidates generated in the process of S64 (S66).

[0138] The determination unit 46 inputs the feature map generated in the process of S66 into the pre-learning model M0 and obtains an output (S67).

[0139] The determination unit 46 determines whether or not the probability value (output result) obtained as a result of the process of S67 matches the label (S68).

[0140] If the output result differs from the label (S68; no), the update unit 47 updates the parameters of the pre-learning model M0 (S69).

[0141] If the output result matches the label (S68; yes), or after the process of S69, the determination unit 46 determines whether the evaluation function at the time of parameter update satisfies the condition (S70).

[0142] If the evaluation function does not satisfy the condition (S70; no), the model learning device 4 selects training data from the training data set T so as not to cause bias in the training data to be selected (S61). Then, the subsequent processes of S62 to S70 are executed. In this way, the processes of S61 to S70 are repeatedly executed until the evaluation function satisfies the condition.

[0143] If the evaluation function satisfies the condition (S70; yes), the update unit 47 outputs the pre-learning model M0 as the determination model M to the beating sound determination device 3 (S71).

[0144] After the processing of S71, the model learning processing ends (END).

[0145] 2.3 Effects of the Modification According to the modification, the feature map generation unit 35 generates multiple feature map candidates, the number of which corresponds to the number of extracted tapping sounds, from multiple spectrograms corresponding to the number of tapping sounds. The feature map generation unit 35 generates a feature map consisting of maximum values ​​in a pair of a corresponding frequency band and a variation frequency band, from multiple feature amounts for each of the multiple feature map candidates. The determination unit 36 ​​determines whether the implant I is inserted properly based on a single output obtained by inputting one feature map into the determination model M. This allows the determination unit 36 ​​to select a sound with a more characteristic (higher) sound pressure from among the multiple tapping sounds. Therefore, as in the embodiment, even if the extracted tapping sounds include sounds resulting from poor tapping, the determination unit 36 ​​can generate the determination result R by excluding such sounds. Therefore, a more reliable determination result R can be presented to the surgeon than when the determination result is calculated from a single tapping sound.

[0146] 17 is a diagram showing an example of evaluation of a determination result using a teacher data set by a determination model generated by a model learning device according to a modified example. In the example of Fig. 17, the accuracy rate, the number of overestimates, and the number of overestimates when determining 42 teacher data sets T using a determination model M generated based on the maximum value of the feature amount are shown for each seed.

[0147] As shown in Figure 17, when the training dataset T was judged using the judgment model M generated based on the maximum value of the feature amount, there were about 1 to 3 over-evaluations, but no oversights. The accuracy rate reached 96%.

[0148] 18 is a diagram showing an example of evaluation of a determination result using measurement data by a determination model generated by a model learning device according to a modified example. In the example of Fig. 18, the accuracy rate, the number of overestimates, and the number of overestimates when 19 pieces of measurement data D are determined by a determination model M generated based on the maximum value of the feature amount are shown for each seed.

[0149] 18, when the measurement data D was judged using the judgment model M generated based on the maximum value of the feature amount, the number of over-evaluations was about 1 to 4, but the number of over-evaluations was only about 0 to 2. The accuracy rate was about 85%.

[0150] As described above, when verification was performed using the configuration according to the modified example, it was found that when the maximum value of the feature quantity is used to generate the determination result R, it is possible to particularly accurately exclude the missed detection cases from the above-mentioned missed detection cases and overestimated cases. Therefore, according to the modified example, it was shown that by generating the determination result R using the maximum value of the feature quantity, it is possible to particularly accurately exclude the missed detection cases.

[0151] 3. Others Various modifications can be applied to the above-described embodiment and modified examples.

[0152] In the above embodiment and modified example, a case where the quality of the tapping sound generated when inserting an implant I into a femur B in total hip replacement surgery has been described, but the present invention is not limited to this. For example, the present invention can be applied to cases where any joint other than a hip joint is replaced with an artificial joint. Furthermore, the present invention can be applied to cases where the quality of the tapping sound generated when inserting an implant into any bone, not limited to artificial joint replacement surgery, has been described.

[0153] In the above embodiment and modified example, the beating sound determination device 3 and the model learning device 4 are separate devices, but this is not limiting. For example, the beating sound determination device 3 and the model learning device 4 may be an integrated device.

[0154] In the above embodiment and modified example, the case where the programs for executing the beating sound determination process and the model learning process are executed by the beating sound determination device 3 and the model learning device 4 has been described, but this is not limiting. For example, the programs for executing the beating sound determination process and the model learning process may be executed by computational resources on the cloud.

[0155] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0156] 1...tapping sound determination system, 2...sensor, 3...tapping sound determination device, 4...model learning device, 11, 21...control circuit, 12, 22...storage, 13, 23...communication module, 14, 24...user interface, 15, 25...drive, 16, 26...storage medium, 31, 41...reverberation sound removal unit, 32, 42...spectrogram generation unit, 33, 43...harmonic sound removal unit, 34, 44...pulling sound removal unit, 35, 45...feature map generation unit, 36, 46...determination unit, 47...update unit, D...measurement data, M...determination model, M0...pre-learning model, R...determination result, H...hammer, I...implant, B...femur.

Claims

1. A tapping sound judgment device comprising a processor configured to execute the following operations: extracting a plurality of partial data corresponding to tapping sounds from time-series sound data obtained when inserting an implant into a bone; calculating, for each of the plurality of partial data, a value of a variable frequency band at each frequency for a frequency band including a plurality of frequencies; generating an input to a model based on the values ​​calculated for the frequency bands of the plurality of partial data; and judging whether the insertion of the implant is successful or not based on an output obtained from the model.

2. The tapping sound judgment device according to claim 1, wherein the plurality of partial data include first partial data, second partial data, and third partial data, wherein the generating step includes generating a first input based on a value calculated for the first partial data, generating a second input based on a value calculated for the second partial data, and generating a third input based on a value calculated for the third partial data, and wherein the judging step includes judging whether the insertion of the implant is successful or not by majority vote of a first output obtained by inputting the first input to the model, a second output obtained by inputting the second input to the model, and a third output obtained by inputting the third input to the model.

3. The beating sound determination device according to claim 1, wherein the generating step includes generating, as the input, a value composed of maximum values ​​in a set of a corresponding frequency band and a variable frequency band from values ​​calculated for the frequency bands of the plurality of partial data.

4. The percussion sound determination device according to claim 1, wherein the extracting step includes: classifying the sound data into harmonic sound data and a plurality of percussion sound data; and removing, from the plurality of percussion sound data, data of a sound produced when the implant is pulled out.

5. The tapping sound determination device according to claim 4, wherein the extracting step further comprises removing echo data from within the operating room from the sound data.

6. The tapping sound determination device according to claim 1, wherein the analysis unit of the value is less than 100 milliseconds.

7. The beating sound determination device according to claim 1, wherein the model is a convolutional neural network.

8. A model learning device comprising a processor configured to perform the following: extracting a plurality of partial data corresponding to tapping sounds from time-series sound data obtained when inserting an implant into a bone, which is training data for a model; calculating, for each of the plurality of partial data, a value of a variable frequency band at each frequency for a frequency band including a plurality of frequencies; generating input to the model based on the values ​​calculated for the frequency bands of the plurality of partial data; determining whether the insertion of the implant is successful or not based on the output obtained from the model; and updating the model based on the values ​​calculated for the frequency bands of the plurality of partial data and labels corresponding to the sound data.

9. A method for determining a tapping sound, comprising: extracting a plurality of partial data corresponding to a tapping sound from time-series sound data obtained when inserting an implant into a bone; calculating, for each of the plurality of partial data, a value of a variable frequency band at each frequency for a frequency band including a plurality of frequencies; generating an input to a model based on the values ​​calculated for the frequency bands of the plurality of partial data; and determining whether the insertion of the implant is successful or not based on an output obtained from the model.

10. A program for causing a processor to execute the following steps: extracting a plurality of partial data corresponding to tapping sounds from time-series sound data obtained when inserting an implant into a bone; calculating, for each of the plurality of partial data, the value of a variable frequency band at each frequency for a frequency band including a plurality of frequencies; generating input to a model based on the values ​​calculated for the frequency bands of the plurality of partial data; and determining whether the insertion of the implant is successful or not based on the output obtained from the model.

Images