Sound output method

Abstract

To provide a sound output method which enables a user to easily enjoy brushing teeth.SOLUTION: A sound output method is implemented by a computer and includes acquiring a captured moving image that is a moving image obtained by capturing a user who brushes teeth, detecting a cycle in which a position of a hand of the user is repeated in a predetermined direction in the captured moving image, and outputting a predetermined sound in accordance with the detected cycle.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a sound output method. [Background technology]

[0002] Conventionally, there has been known a technique for playing music while brushing teeth to make the tooth brushing experience more enjoyable for the user. For example, Patent Document 1 discloses a technique for playing music while brushing teeth using a playback device provided in a toothbrush. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-006937 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with this technology, the electrical circuitry, including the power source and playback device, is installed on the toothbrush itself, which increases the overall size and weight of the toothbrush and makes it less user-friendly. While this technology is useful for children who have difficulty brushing their teeth, installing electrical circuits in a small toothbrush for children makes it more difficult to use than an adult toothbrush.

[0005] Furthermore, the above technology requires the user to prepare music in advance because music that has been pre-recorded on a recording device or the like attached to the toothbrush is played.Furthermore, the toothbrush simply plays pre-recorded music, which limits the enjoyment of tooth brushing.

[0006] In view of the above circumstances, the present invention provides a sound output method that allows a user to easily enjoy brushing their teeth. [Means for solving the problem]

[0007] A sound output method according to one aspect of the present invention is implemented by a computer, and involves acquiring a video of a user brushing their teeth, detecting a cycle in which the position of the user's hand repeats in a predetermined direction in the video, and outputting a predetermined sound according to the detected cycle. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Figure 2] 10 is a flowchart illustrating an example of processing by a user terminal. [Figure 3] FIG. 10 is a diagram illustrating an example of an application screen displayed on a user terminal. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described with reference to the accompanying drawings.

[0010] The information processing system 1 includes a user terminal 10 and a server 20. The components included in the information processing system 1 are connected to each other so that they can communicate with each other via a network 30. The user terminal 10 is a computer such as a smartphone, tablet, or PC used by a user. The server 20 provides predetermined services, apps, etc. to the user terminal 10.

[0011] The user terminal 10 includes a control unit 11, a memory unit 12, a communication unit 13, a display unit 14, an operation unit 15, a photographing unit 16, and a sound output unit 17. The control unit 11 includes, for example, a CPU and executes various processes and controls according to a program. The memory unit 12 includes, for example, a storage device, a ROM, a RAM, etc., and stores various programs and data. The communication unit 13 includes a communication interface and communicates with other devices such as a server 20 via a network 30. The display unit 14 includes, for example, an LCD display or an organic EL display, etc., and displays a screen containing various information, images, etc. The operation unit 15 includes, for example, physical keys or a touch panel integrated with the display unit 14, etc., and is operated by a user. The photographing unit 16 includes a camera and captures images including videos. The sound output unit 17 includes a speaker and outputs sound.

[0012] The server 20 includes a control unit 21, a storage unit 22, and a communication unit 23. The components of the server 20 are the same as those of the user terminal 10, and therefore a description thereof will be omitted.

[0013] The following describes the processing of the user terminal 10. Fig. 2 is a flowchart showing an example of the processing of the user terminal 10. Fig. 3 is a diagram showing an example of an application screen displayed on the user terminal 10.

[0014] It is assumed that a toothbrushing application (hereinafter referred to as "app") is pre-installed on the user terminal 10. The toothbrushing app may be downloaded and installed on the user terminal 10 from a server 20 that manages an app store via a network 30, for example. The control unit 11 of the user terminal 10 executes the process shown in FIG. 2 in accordance with a program included in the toothbrushing app stored in the storage unit 12.

[0015] First, when the control unit 11 receives an operation from a user to start using the toothbrushing app via the operation unit 15, the control unit 11 causes the image capturing unit 16 to start capturing a video of the user brushing their teeth (step S1). For example, the image capturing unit 16 may be an in-camera 161 of the user terminal 10 as shown in FIG. 3 so that the user can brush their teeth while looking at the display unit 14.

[0016] The control unit 11 acquires the moving image whose shooting has started in step S1 (i.e., the moving image being shot, hereinafter referred to as the "shot moving image") (step S2). Then, the control unit 11 causes the display unit 14 to display the shot moving image acquired in step S2, as shown in FIG. 3 (step S3).

[0017] The control unit 11 detects the position of the user's hand holding the toothbrush in the captured video acquired in step S2 (step S4). Then, the control unit 11 detects a cycle in which the detected hand position repeats in a predetermined direction (step S5). Note that the control unit 11 executes various processes shown in Fig. 3, including the process of detecting the cycle in the captured video, in real time while the user is brushing their teeth.

[0018] The predetermined direction may be, for example, the x-axis direction (horizontal or left-right direction) as shown in FIG. 3 , in which case the control unit 11 may detect the user's horizontal brushing cycle. The predetermined direction may also be the y-axis direction (vertical or up-down direction), in which case the control unit 11 may detect the user's vertical brushing cycle. The control unit 11 may detect cycles in multiple predetermined directions, for example, in both the x-axis direction and the y-axis direction. The x-axis and y-axis directions are not limited to the example shown in FIG. 3 , and the control unit 11 does not need to determine the horizontal direction of the captured image as the x-axis and the vertical direction as the y-axis. For example, the control unit 11 may further detect the position and angle of the user's face and determine the horizontal direction of the user's face as the x-axis and the vertical direction as the y-axis, thereby more accurately detecting the user's horizontal and vertical brushing cycles.

[0019] Control unit 11 determines the tone of the sound to be output by sound output unit 17 (step S6). Then, control unit 11 controls sound output unit 17 to repeatedly output the sound of the tone determined in step S6 in accordance with the period detected in step S5 (step S7). Hereinafter, the sound that is repeatedly output is referred to as a "repeated sound." Control unit 11 controls sound output unit 17 to output a short repetitive sound (e.g., a beat) for each detected period.

[0020] Various patterns can be considered for the tone of the repeated sound determined in step S6. For example, control unit 11 may control sound output unit 17 to output a different tone for each direction in which the period was detected in step S5. Control unit 11 may determine to output a repeated sound of a first tone (e.g., a low tone) when the period in the x-axis direction (horizontal brushing) is detected, and to output a repeated sound of a second tone (e.g., a high tone) when the period in the y-axis direction (vertical brushing) is detected.

[0021] In addition to the position of the user's hand holding the toothbrush, the control unit 11 may further detect the angle of the user's hand and the position and angle of the user's face. The control unit 11 may then estimate the position of the teeth the user is currently brushing based on the detected position and angle of the user's hand and the position and angle of the user's face, and determine to output a repetitive sound with a tone corresponding to the estimated tooth position. For example, if the control unit 11 estimates that the teeth being brushed are the front teeth, it may output a repetitive sound with a third tone, and if the teeth being brushed are the back teeth, it may output a repetitive sound with a fourth tone. The control unit 11 may also gradually change the tone of the repetitive sound as the position of the teeth being brushed moves from the front teeth to the back teeth, or from the back teeth to the front teeth. If the position of the toothbrush head can be directly detected in the captured video, the control unit 11 may identify the position of the teeth the user is brushing based on the detected position of the head. The control unit 11 may estimate the position of the teeth the user is brushing based on the position and angle of the user's hand and the position and angle of the user's face only when the toothbrush head is inside the user's mouth in the captured video and the position of the head cannot be detected.

[0022] Furthermore, control unit 11 may detect the position of the user's hand (e.g., left hand) not holding the toothbrush, in addition to the position of the user's hand (e.g., right hand). Then, in step S6, control unit 11 may determine the timbre of the repeated sound depending on the state (e.g., hand movement) of the hand not holding the toothbrush. Furthermore, if the hand not holding the toothbrush is holding a cup, toothpaste, or the like, control unit 11 may determine the timbre of the repeated sound depending on the type of object being held. Furthermore, control unit 11 may detect the position of the user's mouth in addition to the position of the user's hand holding the toothbrush, and may determine the timbre of the repeated sound depending on the state (e.g., the size and shape of the mouth opening) of the user.

[0023] Furthermore, when the detected number of repetitions exceeds a predetermined number or when the duration of the repetitive motion exceeds a predetermined time, control unit 11 may further change the tone of the repetitive sound. Furthermore, control unit 11 may cause sound output unit 17 to output a sound (e.g., a notification sound) with a tone different from the repetitive sound when such a criterion is exceeded.

[0024] Furthermore, control unit 11 may determine the tone of the repeated sound based on any of a plurality of factors including the direction in which the cycle is detected, the position of the teeth being brushed, the state of the user's hand not holding the toothbrush, the state of the user's mouth, the number of repetitions, the duration of the repetitive motion, etc. For example, control unit 11 may control sound output unit 17 to output different sounds when brushing front teeth vertically, when brushing back teeth vertically, when brushing front teeth horizontally, and when brushing back teeth horizontally.

[0025] Various patterns are also possible for the timing of outputting the repetitive sound in step S7. For example, control unit 11 may control sound output unit 17 to output a repetitive sound not at every detected period, but at every predetermined multiple of the period (for example, two periods). Control unit 11 may also change the repetitive sound at every predetermined multiple of the period. For example, control unit 11 may control sound output unit 17 to output a strong (high volume) repetitive sound at every odd multiple of the period and to output a weak (low volume) repetitive sound at every even multiple of the period (i.e., to alternately output repetitive sounds of different intensities).

[0026] Next, control unit 11 causes display unit 14 to display various pieces of information related to tooth brushing (step S8). Control unit 11 causes display unit 14 to display, for example, information corresponding to the repetitive sound output in step S7. Control unit 11 may control display unit 14 to display information on BPM (Beats Per Minute) corresponding to the period of the repetitive sound superimposed on the captured video, as shown in the upper part of the tooth brushing app screen in FIG. 3. Control unit 11 may also control display unit 14 to illuminate or change color at least a portion of the captured video being displayed (for example, an area of ​​the user's teeth) in synchronization with the output timing of the repetitive sound. Control unit 11 may also control display unit 14 to further change the display when the number of repetitions exceeds a predetermined number or when the duration of the repetitive action exceeds a predetermined time.

[0027] Furthermore, when control unit 11 has estimated the positions of the teeth that the user is brushing as described above, it may manage the progress of tooth brushing based on the estimation result and cause display unit 14 to display information indicating the progress of tooth brushing, as shown in the lower right corner of the tooth brushing app screen in Fig. 3. Furthermore, when control unit 11 estimates that a predetermined percentage (or a predetermined area) or more of the teeth have been brushed, it may notify the user that the end of tooth brushing is approaching by causing sound output unit 17 to output predetermined music, causing display unit 14 to display a predetermined message, or changing at least a portion of the display, for example.

[0028] The control unit 11 determines whether the user has finished brushing their teeth (step S9). The control unit 11 repeats the processes of steps S1 to S8 until it determines that the user has finished brushing their teeth (step S9: NO). When it determines that the user has finished brushing their teeth (step S9: YES), it ends the processing of the toothbrushing app, including the photographing process. The control unit 11 may determine that the user has finished brushing their teeth, for example, when it receives a user's operation to end use of the toothbrushing app, when it is no longer possible to detect the repeating cycle of the user's hand position, or when it detects that the user's hand has released the toothbrush. Furthermore, when the control unit 11 has been managing the progress of toothbrushing as described above, it may determine that the user has finished brushing their teeth when it determines that all teeth have been brushed. The control unit 11 may notify the user that the user has finished brushing their teeth by causing the sound output unit 17 to output predetermined music praising the user for finishing brushing their teeth, by causing the display unit 14 to display a predetermined message, or by changing at least a portion of the display.

[0029] As described above, in this embodiment, a video of a user brushing their teeth is captured, and a predetermined sound is output in accordance with the frequency of the user's hand position. This allows users to easily enjoy brushing their teeth without the need for tedious preparations such as preparing a dedicated toothbrush equipped with an electrical circuit and recording existing music on the toothbrush. Furthermore, unlike when existing music is simply played, this new way of enjoying tooth brushing, in which sounds are generated by the user's own movements, can be provided to a variety of users, from children who dislike brushing their teeth to adults who dislike brushing their teeth.

[0030] In this embodiment, it has been described that control unit 11 detects the period in step S5. However, instead of or in addition to detecting the period, control unit 11 may detect the frequency, which is the reciprocal of the period. Control unit 11 may then output a repetitive sound in accordance with the detected frequency. For example, control unit 11 may determine in step S6 to output a sound having a frequency that is a predetermined multiple of the detected frequency, or may control the output timing of the repetitive sound based on the detected frequency in step S7.

[0031] Although the present embodiment has been described with reference to an example in which a user uses the toothbrushing app alone, a service may be provided in which multiple users (e.g., family members or friends) use the toothbrushing app simultaneously and can listen to each other's generated sounds on the toothbrushing app. In this case, the multiple users can enjoy a toothbrushing session using the sounds each other generates.

[0032] Furthermore, in this embodiment, a predetermined sound is output in accordance with the cycle of the repetition of the position of the user's hand, but a service may be provided in which a predetermined sound is output in accordance with the cycle of the repetition of the position of any object other than the user's hand. For example, a service may be provided in which a predetermined sound is output in accordance with the cycle of the repetition of the position of the user's head or nose up and down or left and right. Furthermore, a service may be provided in which a predetermined sound is output in accordance with the cycle of any repetitive action that is enjoyable in daily life when it makes a sound (for example, the action of a person moving a vacuum cleaner back and forth, or the action of a cat scratching its claws, etc.). [Explanation of symbols]

[0033] 1: Information processing system 10: User terminal 11, 21: Control unit 12, 22: Storage section 13, 23: Communications Department 14: Display section 15:Operation unit 16: Photography Department 161: In-camera 17: Sound output section 20: Server 30: Network

Claims

[Claim 1] 1. A computer-implemented sound output method, comprising: Acquire a captured video that is a video of a user brushing their teeth; Detecting a cycle in which the position of the user's hand repeats in a predetermined direction in the captured video; outputting a predetermined sound in accordance with the detected period; Sound output method.

Images