Playback control method, playback control system, and program

Abstract

To provide a reproduction control method, a reproduction control system and a program that reduce a burden on a user in an instruction about reproduction of a sound.SOLUTION: In a reproduction control system 100, a control unit 10 comprises a reproduction control part 31 which causes a sound emission device to reproduce a target sound corresponding to a detection unit according to a state of a hand on an operation surface of a detection unit by detection signals Q1, Q2 detected by a first detector and a second detector, and changes a feature quantity of the sound according to a parameter related to a position of an object in a process where the object moves in a state apart from the operation surface. The first detector is an optical sensor which detects a state of the hand H of a user, and generates the detection signal Q1 representing a distance from a light reception surface to the hand H in time series, and the second detector is a sensor for detecting touch of the hand with respect to the operation surface and generates the detection signal Q2 representing an ambient sound including a strike sound.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to technology for controlling sound. [Background technology]

[0002] For example, various techniques have been proposed for changing the feature quantities of sounds that are reproduced in response to instructions from a user. For example, Patent Document 1 discloses an electronic musical instrument that changes (pitch bends) the pitch of sounds that are reproduced in response to key depressions by the user in response to the user's operation of a pitch bend wheel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-161699 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional configurations, the user must operate the pitch bend wheel to instruct pitch bend separately from instructing sound generation by pressing a key. This poses a problem of a heavy burden on the user to perform operations to reproduce sound. In consideration of the above circumstances, one aspect of the present disclosure aims to reduce the burden on the user of instructions regarding sound reproduction. [Means for solving the problem]

[0005] In order to solve the above problems, a playback control method according to one aspect of the present disclosure plays a sound when an object is in contact with an operation surface, and changes the characteristics of the sound according to parameters related to the position of the object as the object moves away from the operation surface.

[0006] A playback control system according to one aspect of the present disclosure includes a playback control unit that plays a sound when an object is in contact with an operation surface, and changes a characteristic amount of the sound in accordance with a parameter related to the position of the object as the object moves away from the operation surface.

[0007] A program according to one aspect of the present disclosure causes a computer to function as a playback control unit that plays a sound when an object is in contact with an operation surface, and changes the characteristics of the sound in accordance with parameters related to the position of the object as the object moves away from the operation surface. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of a playback control system. [Figure 2] FIG. 2 is a schematic diagram illustrating the configuration of a detection unit. [Figure 3] FIG. 2 is a block diagram illustrating an example of the functional configuration of the control system. [Figure 4] FIG. 10 is an explanatory diagram regarding the state of a user's hand. [Figure 5] 10 is a flowchart illustrating a specific procedure of a control process. [Figure 6] FIG. 11 is an explanatory diagram regarding the state of the hand in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] A: First embodiment FIG. 1 is a block diagram illustrating the configuration of a playback control system 100 according to a first embodiment of the present disclosure. The playback control system 100 is a computer system that plays a sound (hereinafter referred to as a "target sound") in response to a user's operation. The playback control system 100 includes a control system 1 and multiple detection units 2. The multiple detection units 2 detect operations by the user. The control system 1 plays a target sound in response to the operation detected by the detection units 2. The target sound played by the control system 1 is, for example, a sound played on an instrument such as a keyboard instrument. However, sounds such as singing or speaking may also be played as the target sound.

[0010] The control system 1 includes a control device 10, a storage device 11, and a sound emitting device 13. The control system 1 is realized by an information terminal such as a smartphone, a tablet terminal, or a personal computer. The control system 1 may be realized by a single device, or may be realized by multiple devices configured separately from each other.

[0011] The control device 10 is one or more processors that control each element of the control system 1. Specifically, the control device 10 is configured by one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit). The control device 10 generates an acoustic signal X that represents the waveform of a target sound in response to a user operation.

[0012] The sound emitting device 13 reproduces a target sound represented by the acoustic signal X generated by the control device 10. The sound emitting device 13 is, for example, a speaker or headphones. For convenience, a D / A converter that converts the acoustic signal X from digital to analog and an amplifier that amplifies the acoustic signal X are not shown in the figure. Furthermore, while FIG. 1 illustrates a configuration in which the sound emitting device 13 is mounted on the control system 1, the sound emitting device 13 may be separate from the control system 1 and connected to the control system 1 by wire or wirelessly.

[0013] The storage device 11 is one or more memories that store programs executed by the control device 10 and various data used by the control device 10. The storage device 11 is configured from a known storage medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of storage media. Note that a storage device 11 (e.g., cloud storage) separate from the control system 1 may be provided, and the control device 10 may write and read data to and from the storage device 11 via a communication network such as a mobile communication network or the Internet. In other words, the storage device 11 may be omitted from the control system 1.

[0014] The playback control system 100 includes a plurality of detection units 2 corresponding to different pitches (hereinafter referred to as "standard pitches") Ps. Each of the plurality of detection units 2 is an operator that allows a user to instruct the reproduction of a target sound of the standard pitch Ps corresponding to that detection unit 2. When the user operates a detection unit 2 among the plurality of detection units 2 that corresponds to a desired standard pitch Ps, the target sound of that standard pitch Ps is reproduced. Each detection unit 2 constitutes, for example, a key of a keyboard instrument. In other words, an arrangement of the plurality of detection units 2 constitutes a keyboard, and the playback control system 100 is realized as a keyboard instrument.

[0015] FIG. 2 is a schematic diagram illustrating the configuration of any one of the detection units 2. The detection unit 2 includes a housing 20, a first detector 21, and a second detector 22. The housing 20 in FIG. 2 is a hollow structure that houses the first detector 21 and the second detector 22. Specifically, the housing 20 includes a housing 20a and a light-transmitting portion 20b. The housing 20a is a hollow box-shaped structure that is open at the top. The light-transmitting portion 20b is a plate-like member that closes the opening of the housing 20a. The light-transmitting portion 20b transmits light in a wavelength range that can be detected by the first detector 21. A user can move their hand H toward or away from a surface F of the light-transmitting portion 20b opposite the housing 20a (hereinafter referred to as the "operation surface") and strike the operation surface F with their hand H. The user's hand H is an example of an "object."

[0016] The first detector 21 is an optical sensor that detects the state of the user's hand H. The first detector 21 is installed near the midpoint on the bottom surface of the housing 20a. Specifically, a distance measuring sensor that measures the distance between an object and a light receiving surface is used as the first detector 21. For example, the first detector 21 receives reflected light from the hand H that passes through the light-transmitting portion 20b, and generates a detection signal Q1 that time-series represents the position of the hand H in a direction perpendicular to the operation surface F (specifically, the distance from the light receiving surface to the hand H). The detection signal Q1 is transmitted to the control system 1 via wired or wireless communication. Note that the light detected by the first detector 21 is not limited to visible light. For example, the first detector 21 may receive invisible light such as infrared light.

[0017] The second detector 22 is a sensor for detecting contact of the hand H with the operation surface F. For example, a sound collection device that collects ambient sounds is used as the second detector 22. The second detector 22 collects the impact sound generated when the user's hand H strikes the operation surface F. The second detector 22 generates a detection signal Q2 that represents the ambient sounds including the impact sound. The detection signal Q2 is transmitted to the control system 1 via wired or wireless communication. The second detector 22 may be installed outside the housing 20.

[0018] 3 is a block diagram illustrating an example of the functional configuration of the control system 1. The control device 10 of the control system 1 executes a program stored in the storage device 11 to realize multiple functions (a state detection unit 30 and a playback control unit 31).

[0019] The state detection unit 30 detects the state of the user's hand H according to the detection results (detection signal Q1 and detection signal Q2) of each of the multiple detection units 2. Specifically, the state detection unit 30 detects a first state or a second state as the state of the hand H. As illustrated in FIG. 2, the first state is a state in which the hand H is separated from the operation surface F of one detection unit 2 by a predetermined distance (hereinafter referred to as "reference value") Dref. The second state is a state in which the hand H is in contact with the operation surface F.

[0020] FIG. 4 is an explanatory diagram of the state of the hand H. The state detection unit 30 detects that the hand H is in the first state by analyzing the detection signal Q1 generated by the first detector 21. Specifically, the state detection unit 30 calculates the distance D between the operation surface F and the hand H by analyzing the detection signal Q1. The calculation of the distance D is repeated at a predetermined cycle. That is, a time series of the distance D is generated. Any known technique may be used to calculate the distance D. FIG. 4 illustrates the change in the distance D over time. The state detection unit 30 determines that the hand H is in the first state when the distance D between the operation surface F and the hand H matches a reference value Dref. Alternatively, the state detection unit 30 may determine that the hand H is in the first state when the distance D is within a predetermined tolerance range including the reference value Dref. The reference value Dref is a fixed value set in advance. However, the reference value Dref may be changed in response to an instruction from the user.

[0021] Furthermore, the state detection unit 30 detects that the hand H is in the second state by analyzing the detection signal Q2 generated by the second detector 22. Specifically, the state detection unit 30 calculates the volume V of the sound represented by the detection signal Q2. The calculation of the volume V is repeated at a predetermined cycle. That is, a time series of the volume V is generated. Any known technology may be used to calculate the volume V. FIG. 4 illustrates the temporal change in the volume V. When an impact on the operation surface F generates an impact sound, the volume V increases sharply. The state detection unit 30 determines that the hand H is in the second state when the volume V exceeds a predetermined value (hereinafter referred to as the "reference value") Vref (i.e., when an impact sound is picked up). The reference value Vref is a fixed value set in advance. However, the reference value Vref may be changed in response to an instruction from the user.

[0022] A user can instruct the reproduction of a target sound of a desired standard pitch Ps by bringing a hand H close to the operation surface F of one of the plurality of detection units 2 that corresponds to the desired standard pitch Ps. The user's hand H sequentially changes from a first state to a second state during a series of processes in which the hand H approaches the operation surface F of one of the detection units 2. Specifically, at a specific time point t1 (hereinafter referred to as the "first time point") during the process in which the hand H approaches the operation surface F, the hand H changes to the first state, and at a time point t2 (hereinafter referred to as the "second time point") after the first time point t1 has passed, the hand H changes to the second state.

[0023] The first time point t1 and the second time point t2 are spaced apart on the time axis. FIG. 4 illustrates a time point t3 (hereinafter referred to as the "third time point") after the second time point t2. The third time point t3 is a time point a predetermined length of time has elapsed since the second time point t2. FIG. 4 also illustrates a first period T1 and a second period T2. The first period T1 is the period from the first time point t1 to the second time point t2, and the second period T2 is the period from the second time point t2 to the third time point t3. The length of the first period T1 (the interval between the first time point t1 and the second time point t2) varies depending on the speed at which the user moves the hand H. The length of the second period T2 (the interval between the second time point t2 and the third time point t3) may be changed, for example, in response to an instruction from the user.

[0024] The playback control unit 31 in FIG. 3 causes the sound emitting device 13 to play a target sound corresponding to each detection unit 2 in accordance with the state of the hand H relative to the operation surface F of that detection unit 2. The target sound of standard pitch Ps corresponding to the detection unit 2 that the user's hand H approaches among the multiple detection units 2 is played. Specifically, the playback control unit 31 generates an audio signal X representing the target sound. For example, the storage device 11 stores multiple waveform data representing waveforms of sounds of different standard pitches Ps. The playback control unit 31 reads from the storage device 11 the waveform data of standard pitch Ps that corresponds to the detection unit 2 that the user's hand H approaches among the multiple detection units 2, and processes the waveform data to generate the audio signal X. The audio signal X is supplied to the sound emitting device 13, whereby the target sound is played.

[0025] 4, when the user's hand H approaches the operation surface F of any of the plurality of detection units 2, the reproduction control unit 31 generates the acoustic signal X so that reproduction of the target sound starts at a first time point t1 and continues from the first time point t1 to a third time point t3 after the second time point t2 has elapsed. In other words, the target sound continues to be emitted from the first time point t1 before the hand H comes into contact with the operation surface F to the third time point t3 after the contact.

[0026] The playback control unit 31 changes the pitch P of the target sound over time within a first period T1 from a first time point t1 to a second time point t2. The playback control unit 31 controls the change in pitch P over time within the first period T1 (i.e., the trajectory of the change in pitch P along the time axis). Specifically, the playback control unit 31 changes the pitch P of the target sound linearly or curvedly within the first period T1 from a first pitch P1 to a standard pitch Ps. The first pitch P1 is a pitch that is lower than the standard pitch Ps by a predetermined value. Therefore, the first pitch P1 differs for each detection unit 2. Specifically, the playback control unit 31 changes the pitch P of the target sound throughout the entire first period T1 so that the pitch P of the target sound becomes the first pitch P1 at the first time point t1 and reaches the standard pitch Ps at the second time point t2. The change in pitch P during the first period T1 corresponds to a pitch bend of the target sound. The pitch P is an example of a "feature quantity" of the target sound, and the standard pitch Ps is an example of a "target value."

[0027] To achieve the above-described change in pitch P, the playback control unit 31 changes the pitch P of the target sound at a speed corresponding to the moving speed of the user's hand H during the first period T1 (hereinafter referred to as the "moving speed"). The moving speed is the amount of change in the distance D per unit time calculated by the state detection unit 30. The state detection unit 30 calculates the moving speed by analyzing the image signal Q1. Specifically, the playback control unit 31 changes the pitch P during the first period T1 so that the faster the moving speed, the faster the speed of change in pitch P. With the above configuration, the user can adjust the speed of change in pitch P during the first period T1 according to the moving speed of the hand H. In the first embodiment, the first pitch P1 and the standard pitch Ps are determined in advance. The playback control unit 31 controls the trajectory (speed of change) of the pitch P from the first pitch P1 to the standard pitch Ps during the first period T1 according to the moving speed of the user's hand H. It is also possible to configure the relationship between the speed of movement within the first period T1 and the speed of change of the pitch P to be changeable in response to an instruction from the user.

[0028] Furthermore, the reproduction control unit 31 maintains the pitch P of the target sound at the standard pitch Ps during a second period T2 from the second time point t2 to the third time point t3. Specifically, the pitch P of the target sound is fixed at the standard pitch Ps throughout the entire second period T2.

[0029] 5 is a flowchart illustrating the specific steps of the process (hereinafter referred to as the "control process") Sa executed by the control device 10. For example, the control process Sa is executed in parallel or sequentially for each of the multiple detection units 2. The control process Sa is repeated at a cycle that is sufficiently shorter than the cycle at which the user's hand H approaches and moves away from the operation surface F.

[0030] When the control process Sa is started, the state detection unit 30 detects the state of the user's hand H by analyzing the detection signals Q1 and Q2 supplied from the detection unit 2 (Sa1). The playback control unit 31 determines whether the state detection unit 30 has detected that the hand H is in a first state (Sa2). If the first state is detected (Sa2: YES), the playback control unit 31 causes the sound emitting device 13 to start playing a target sound of a first pitch P1 corresponding to the standard pitch Ps of the detection unit 2 (Sa3). If the first state is not detected (Sa2: NO), the playback control unit 31 ends the control process Sa.

[0031] When playback of the target sound begins, the playback control unit 31 changes the pitch P of the target sound toward the standard pitch Ps (Sa4). Specifically, the control device 10 brings the pitch P of the target sound closer to the standard pitch Ps by an amount corresponding to the moving speed of the user's hand H. The playback control unit 31 determines whether the state detection unit 30 has detected that the hand H is in the second state (Sa5). If the second state is not detected (Sa5: NO), the playback control unit 31 proceeds to step Sa4. That is, during the first period T1 until the second time point t2 at which the second state is detected, the pitch P of the target sound changes over time toward the standard pitch Ps. Through the above process, the pitch P of the target sound reaches the standard pitch Ps at the second time point t2 at which the second state is detected.

[0032] If the second state is detected (Sa5: YES), the playback control unit 31 maintains the pitch P of the target sound at the standard pitch Ps (Sa6). The playback control unit 31 determines whether the third point in time t3 has arrived (Sa7). The pitch P of the target sound is maintained at the standard pitch Ps until the third point in time t3 arrives (Sa7: NO). That is, the pitch P of the target sound is maintained at the standard pitch Ps during the second period T2. When the third point in time t3 arrives (Sa7: YES), the playback control unit 31 stops the playback of the target sound (Sa8).

[0033] As explained above, in the first embodiment, the target sound continues to be reproduced from the first time point t1, when the hand H is in the first state, to the third time point t3, after the second time point t2, when the hand H is in the second state, while the pitch P of the target sound changes within the first period T1, from the first time point t1 to the second time point t2. Therefore, the user can change the pitch P of the target sound within the first period T1 by simply bringing the hand H closer to the operation surface F. In other words, the burden on the user can be reduced compared to a configuration in which the user needs to separately instruct the reproduction of the target sound and the change in pitch P.

[0034] Furthermore, in the first embodiment, the pitch P of the target sound reaches the standard pitch Ps at the second time point t2 when the user's hand H is in the second state of contacting the operation surface F, which has the advantage that the user can easily indicate the time point at which the pitch P reaches the standard pitch Ps. Furthermore, since the pitch P of the target sound is maintained at the standard pitch Ps during the second period T2 from the second time point t2 to the third time point t3, there is also the advantage that the user can easily indicate playback of the target sound at the standard pitch Ps.

[0035] B: Second embodiment The second embodiment will be described below. Note that, in each of the following exemplary embodiments, for elements whose functions are similar to those of the first embodiment, the reference numerals used in the description of the first embodiment will be used and detailed descriptions of each will be omitted as appropriate.

[0036] The playback control system 100 of the second embodiment detects a state (hereinafter referred to as a "third state") in which the hand H that has been in contact with the operation surface F begins to move away from the operation surface F. Specifically, the state detection unit 30 detects, as the third state, a state in which the distance D between the operation surface F and the hand H begins to increase from zero.

[0037] In the first embodiment, the point in time when a predetermined time has elapsed from the second point in time t2 is designated as the third point in time t3, and the reproduction of the target sound is stopped. In the second embodiment, the point in time when the state detection unit 30 detects the third state is designated as the third point in time t3, and the reproduction of the target sound is stopped. That is, while the user's hand H is in contact with the operation surface F, the reproduction of the target sound of the standard pitch Ps is maintained, and when the hand H is removed from the operation surface F, the reproduction of the target sound is stopped.

[0038] The second embodiment also achieves the same effects as the first embodiment. Furthermore, the second embodiment has the advantage that the user can easily instruct the stopping of the reproduction of the target sound by moving the hand H away from the operation surface F.

[0039] C: Third embodiment 6 is an explanatory diagram of the reproduction of the target sound in the third embodiment. The state detection unit 30 in the third embodiment detects the third state in which the hand H is separated from the operation surface F, similarly to the second embodiment.

[0040] FIG. 6 illustrates a time point t4 (hereinafter referred to as the "fourth time point") after the time point t3 at which the third state was detected. The fourth time point t4 is the time at which a predetermined length of time has elapsed since the third time point t3. The third period T3 in FIG. 6 is the period from the third time point t3 to the fourth time point t4. The third period T3 corresponds to the process in which the hand H moves away from the operation surface F (the process in which the distance D increases). Note that the length of the third period T3 (the interval between the third time point t3 and the fourth time point t4) may be changed, for example, in response to an instruction from the user.

[0041] As in the first embodiment, the playback control unit 31 of the third embodiment maintains the pitch P of the target sound, whose playback begins at the first time point t1, at the standard pitch Ps during the second period T2, and also changes the pitch P of the target sound over time during the third period T3. Specifically, the playback control unit 31 changes the pitch P of the target sound from the standard pitch Ps to a second pitch P2 in a linear or curved manner during the third period T3. The second pitch P2 is a pitch that is lower than the standard pitch Ps by a predetermined value. Therefore, the second pitch P2 differs for each detection unit 2. Specifically, the playback control unit 31 changes the pitch P of the target sound throughout the third period T3 so that the pitch P of the target sound, which is at the standard pitch Ps at the third time point t3, reaches the second pitch P2 at the fourth time point t4. Note that the first pitch P1 and the second pitch P2 may be different or the same, and the pitch may be higher or lower.

[0042] In order to achieve the change in pitch P within the third period T3 described above, the playback control unit 31 changes the pitch P of the target sound at a speed that corresponds to the moving speed of the hand H within the third period T3. Specifically, the playback control unit 31 changes the pitch P within the third period T3 so that the faster the moving speed, the faster the rate of change in pitch P. With the above configuration, the user can adjust the rate of change in pitch P within the third period T3 according to the moving speed of the hand H. Note that a configuration is also envisioned in which the relationship between the moving speed and the rate of change in pitch P within the third period T3 can be changed in response to an instruction from the user.

[0043] D: Modification Specific modified embodiments that can be added to each of the embodiments exemplified above are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined as appropriate within the scope of not being mutually contradictory.

[0044] (1) In the above-described embodiments, a distance measurement sensor is exemplified as the first detector 21, but the type of the first detector 21 is not limited to the above examples. For example, an image sensor that captures an image of the user's hand H may be used as the first detector 21. The state detection unit 30 calculates the distance D by analyzing the image of the hand H captured by the first detector 21, and detects the first state based on the distance D. Also, an infrared sensor that emits and receives infrared light may be used as the first detector 21. The state detection unit 30 calculates the distance D from the received light intensity of the infrared light reflected by the surface of the hand H. Also, the first detector 21 may be installed in any position. For example, the first detector 21 may capture an image of the hand H from the side.

[0045] (2) In the above-described embodiments, the second state is detected by analyzing the detection signal Q2 representing a sound including a striking sound. However, the configuration and method for detecting the contact of the hand H with the operation surface F are not limited to the above examples. For example, the contact of the hand H with the operation surface F (i.e., the second state) may be detected by analyzing the detection signal Q1 generated by the first detector 21. For example, the state detection unit 30 determines that the user's hand H is in the second state when the distance D determined from the detection signal Q1 reaches zero. In a configuration using the detection signal Q1 to detect the second state, the second detector 22 is omitted. Alternatively, the second detector 22 may be a contact sensor (e.g., a capacitance sensor) that detects the contact of the hand H with the operation surface F (translucent portion 20b), a vibration sensor that detects vibrations of the operation surface F (translucent portion 20b), or a pressure sensor that detects the pressure applied by the hand H to the operation surface F.

[0046] (3) In each of the above-described embodiments, a configuration in which the user's hand H contacts the operation surface F has been exemplified, but the object that contacts the operation surface F is not limited to the hand H. For example, the user may strike the operation surface F with a striking member such as a percussion stick. As can be understood from the above examples, the object that contacts the operation surface F includes both a part of the user's body (typically the hand H) and the striking member operated by the user. Note that in a configuration in which a striking member strikes the operation surface F, the first detector 21 or the second detector 22 may be mounted on the striking member.

[0047] (4) The configuration of the housing 20 of the detection unit 2 is arbitrary. In addition, it is not essential that the first detector 21 and the second detector 22 are housed in the housing 20. In other words, as long as the detection unit 2 is configured to include the operation surface F that comes into contact with an object such as the user's hand H, the specific configuration of the housing 20 and whether it is present or not are irrelevant.

[0048] (5) In the above embodiments, the pitch P of the target sound is changed throughout the entire first period T1. However, the pitch P of the target sound may be changed during a portion of the first period T1 and maintained at the standard pitch Ps for the remainder of the period. In other words, the pitch P of the target sound may reach the standard pitch Ps before the second point in time t2 arrives. As can be understood from the above explanation, "changing the pitch P during the first period T1" means changing the pitch P during part or all of the first period T1. Similarly, "changing the pitch P during the third period T3" means changing the pitch P during part or all of the third period T3.

[0049] (6) In the above-described embodiments, the pitch P of the target sound is changed at a speed corresponding to the speed of movement of the user's hand H. However, the method of linking the pitch P to the movement of the user's hand H is not limited to the above examples. For example, the pitch P of the target sound may be changed according to the distance D of the hand H from the operation surface F. The playback control unit 31 changes the pitch P of the target sound over time, for example, so that the pitch P increases as the distance D decreases and reaches the standard pitch Ps when the distance D becomes zero. Within the first period T1, the pitch P may also decrease as the distance D increases.

[0050] The playback control unit 31 may also change the pitch P of the target sound depending on the direction in which the user's hand H moves (hereinafter referred to as the "movement direction"). The state detection unit 30 identifies the movement direction of the hand H relative to the operation surface F by analyzing the image signal Q1. The playback control unit 31 controls the relationship of changes in the pitch P relative to the time axis (i.e., the trajectory of the pitch P over time) depending on the movement direction of the hand H. Specifically, when the angle of the movement direction relative to the operation surface F is within a first range, the playback control unit 31 changes the pitch P so that it follows a first trajectory over time. On the other hand, when the angle of the movement direction relative to the operation surface F is within a second range, the playback control unit 31 changes the pitch P so that it follows a second trajectory different from the first trajectory over time. For example, when the angle between the movement direction and the operation surface F exceeds a predetermined threshold, the playback control unit 31 changes the pitch P sharply over time (first trajectory). On the other hand, when the angle between the movement direction and the operation surface F is below the threshold, the playback control unit 31 changes the pitch P gradually over time (second trajectory).

[0051] As can be seen from the above examples, the playback control unit 31 changes the pitch P of the target sound according to parameters related to the position of the user's hand H. The aforementioned movement speed, distance D, and movement direction are specific examples of parameters related to the position of the hand H. The position of the user's hand H itself is also used as a parameter. For example, the parameters related to the position of the hand H include not only the position of the hand H in a direction perpendicular to the operation surface F (e.g., distance D), but also the position of the hand H in a plane parallel to the operation surface F. The parameters related to the position of the hand H are identified, for example, by analyzing the image signal Q1, as exemplified in each of the above embodiments.

[0052] (7) In the above-described embodiments, the first pitch P1 is lower than the standard pitch Ps. However, the first pitch P1 may be higher than the standard pitch Ps. That is, during the first period T1, the pitch P of the target sound decreases over time from the first pitch P1 to the standard pitch Ps. The first pitch P1 may also be set in response to a user instruction.

[0053] Although the third embodiment illustrates a configuration in which the second pitch P2 is lower than the standard pitch Ps, a configuration in which the second pitch P2 is higher than the standard pitch Ps is also conceivable. That is, during the third period T3, the pitch P of the target sound rises over time from the second pitch P2 to the standard pitch Ps. The second pitch P2 may also be set in response to an instruction from the user.

[0054] (8) In the above-described embodiments, the pitch P of the target sound is controlled in accordance with the state of the hand H. However, the feature of the target sound controlled by the playback control unit 31 is not limited to the pitch P. For example, the volume of the target sound may be controlled in accordance with the state of the user's hand H. Furthermore, the timbre of the target sound may be controlled in accordance with the state of the user's hand H. For example, the playback control unit 31 may mix first waveform data representing a first timbre with second waveform data representing a second timbre to generate an audio signal X representing a target sound with a timbre intermediate between the first and second timbre. The playback control unit 31 changes the mixing ratio of the first waveform data and the second waveform data in accordance with the state of the user's hand H, similar to the pitch P in the above-described embodiments. With the above configuration, for example, within the first period T1 or the third period T3, it is possible to cause the timbre of the target sound to approach one of the first timbre and the second timbre toward the other.

[0055] (9) In the above-described embodiments, a configuration in which the user's hand H actually touches the operation surface F has been exemplified. However, a configuration in which the user touches a virtual operation surface F using, for example, haptics technology that uses tactile feedback may also be employed. The user touches the operation surface F installed in the virtual space by manipulating a pseudo-hand that exists in the virtual space. By using a vibrator that vibrates when touching the operation surface F in the virtual space, the user perceives that they are actually touching the operation surface F. As can be understood from the above explanation, the operation surface F may be a virtual surface in the virtual space. Similarly, an object (e.g., hand H) that touches the operation surface F may be a virtual object in the virtual space.

[0056] (10) As described above, the functions of the playback control system 100 (particularly the functions of the control system 1) are realized by the cooperation of one or more processors constituting the control device 10 and a program stored in the storage device 11. The program according to the present disclosure may be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium may be, for example, a non-transitory recording medium, such as an optical recording medium (optical disc) such as a CD-ROM, but may also include any known type of recording medium, such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium other than a transitory, propagating signal, and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device distributes a program via a communication network, the storage device in the distribution device that stores the program corresponds to the non-transitory recording medium described above.

[0057] E: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0058] A playback control method according to one aspect (aspect 1) of the present disclosure detects a first state in which an object is a predetermined distance away from an operation surface, or a second state in which the object is in contact with the operation surface, starts playing sound at a first point in time when the object reaches the first state during the process of the object approaching the operation surface, continues playing the sound from the first point in time to a third point in time after the second point in time when the object reaches the second state during the process, and controls the change over time in the characteristics of the sound during a first period from the first point in time to the second point in time.

[0059] According to the above aspect, by a simple operation of a user bringing an object close to the operation surface, sound playback continues from a first time point when the object is in a first state to a third time point after the second time point when the object is in a second state, and the feature amount of the sound changes during the first period from the first time point to the second time point. Therefore, the burden on the user can be reduced compared to a configuration in which the user must individually instruct the playback of the sound and the change in the feature amount. The feature amount of the sound is, for example, pitch, volume, or timbre.

[0060] In a specific example (Aspect 2) of Aspect 1, the feature of the sound is changed within the first period so as to reach the target value at the second time point. According to the above aspect, the feature of the sound reaches the target value at the second time point when the object is in the second state of contact with the operation surface, which is advantageous in that the user can easily indicate the time point when the feature of the sound will reach the target value.

[0061] In a specific example (Aspect 3) of Aspect 2, the feature quantity of the sound is maintained at the target value during a second period from the second time point to the third time point. According to the above aspect, since the feature quantity of the sound is maintained at the target value during the second period from the second time point to the third time point, there is an advantage that the user can easily instruct the playback of sound whose feature quantity is the target value.

[0062] In a specific example (Aspect 4) of any one of Aspects 1 to 3, the feature of the sound is changed during the first period according to a parameter related to the position of the object. According to the above aspect, the user can adjust the trajectory of the change in the feature during the first period according to the position of the object. Examples of the parameter related to the position of the object include the speed at which the object moves (Aspect 5), the distance of the object from the operation surface (Aspect 6), or the direction of movement of the object (Aspect 7).

[0063] In a specific example (Aspect 8) of any one of Aspects 1 to 7, the feature of the sound is pitch. With the above configuration, it is possible to control pitch changes (pitch bend) at the beginning (within the first period) when the sound starts to be reproduced.

[0064] A playback control method according to one aspect (aspect 9) of the present invention plays sound when an object is in contact with an operation surface, and changes the characteristics of the sound at a speed corresponding to the speed at which the object moves as the object moves away from the operation surface.

[0065] A playback control system according to one aspect (aspect 10) of the present invention includes a state detection unit that detects a first state in which an object is a predetermined distance away from an operation surface, or a second state in which the object is in contact with the operation surface, and a playback control unit that starts playing sound at a first point in time when the object reaches the first state during the process of the object approaching the operation surface, continues playing the sound from the first point in time to a third point in time after the second point in time when the object reaches the second state during the process, and controls the time change in the characteristics of the sound during a first period from the first point in time to the second point in time.

[0066] A program according to one aspect (aspect 11) of the present invention causes a computer to function as a state detection unit that detects a first state in which an object is a predetermined distance away from an operation surface, or a second state in which the object is in contact with the operation surface, and a playback control unit that starts playing sound at a first point in time when the object reaches the first state during the process of the object approaching the operation surface, continues playing the sound from the first point in time to a third point in time after the second point in time when the object reaches the second state during the process, and controls the time change in the characteristics of the sound during a first period from the first point in time to the second point in time. [Explanation of symbols]

[0067] 100...playback control system, 1...control system, 10...control device, 11...storage device, 13...sound emission device, 2...detection unit, 20...container, 20a...casing portion, 20b...transparent portion, 21...first detector, 22...second detector, 30...status detection portion, 31...playback control portion.

Claims

1. Playing a sound when an object comes into contact with the operating surface; The feature amount of the sound is changed according to the position or speed of the object in a direction perpendicular to the operation surface while the object is moving away from the operation surface. A computer-implemented playback control method.

2. The sound feature is at least one of pitch, volume, and timbre. The playback control method of claim 1.

3. the feature of the sound is a pitch, The change in pitch is a pitch bend The playback control method according to claim 2.

4. The operating surface is at least a part of a musical instrument.

4. The playback control method according to claim 1.

5. The object is a component for a musical instrument.

4. The playback control method according to claim 1.

6. The operation surface is a virtual surface in a virtual space.

6. The playback control method according to claim 1.

7. a playback control unit that plays back a sound when an object is in contact with the operation surface, and changes a feature amount of the sound according to a position or a speed of the object in a direction perpendicular to the operation surface while the object is moving away from the operation surface; A playback control system comprising:

8. a playback control unit that plays back a sound when an object is in contact with the operation surface, and changes a feature amount of the sound according to a position or a speed of the object in a direction perpendicular to the operation surface while the object is moving away from the operation surface; A program that makes a computer function as a

Images