Signal processing device, signal processing method, and program product

Abstract

This technology relates to a signal processing apparatus, method, and program that can intuitively manipulate sound. The signal processing apparatus includes: an acquisition unit for acquiring sensing values ​​indicating the movement of a predetermined part of a user's body or an instrument; and a control unit for performing nonlinear acoustic processing on the acoustic signal based on the sensing values. This technology can be applied to acoustic reproduction systems.

Description

Technical Field

[0001] This technology relates to signal processing devices, signal processing methods and programs, and in particular to signal processing devices, signal processing methods and programs for realizing intuitive operation of sound. Background Technology

[0002] Previously, technologies for manipulating sound based on the user's body movements have been proposed (e.g., see Patent Document 1).

[0003] For example, in Patent Document 1, since the effect processing is performed based on the output waveform of a sensor attached to the user, the sound is changed according to the motion reproduction when the user moves the attached part of the sensor.

[0004] Furthermore, by using such technology, for example, a DJ can change the volume of the sound being reproduced by moving their arm up and down, that is, the sound can be manipulated.

[0005] Reference List

[0006] Patent documents

[0007] Patent Document 1: WO2017 / 061577 Summary of the Invention

[0008] The problem to be solved by the present invention

[0009] However, users find it difficult to intuitively manipulate sound using the aforementioned technologies because even if the sensor's output waveform is directly applied to parameters to manipulate sound, it cannot fully reflect the user's intent when manipulating sound.

[0010] This technology was developed in view of the above circumstances and is designed to enable intuitive operation of sound.

[0011] Solution to the problem

[0012] A signal processing apparatus according to one aspect of the present technology includes: an acquisition unit that acquires sensing values ​​indicating the movement of a predetermined part of a user's body or the movement of an apparatus; and a control unit that performs nonlinear acoustic processing on an acoustic signal based on the sensing values.

[0013] A signal processing method or procedure according to one aspect of the present technology includes the following steps: acquiring sensing values ​​indicating the movement of a predetermined part of a user's body or the movement of an apparatus; and performing nonlinear acoustic processing on an acoustic signal based on the sensing values.

[0014] In one aspect of this technology, sensing values ​​indicating the movement of a predetermined part of a user's body or the movement of an instrument are acquired, and nonlinear acoustic processing is performed on the acoustic signal based on the sensing values. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating an example configuration of an acoustic reproduction system.

[0016] Figure 2 This is a diagram illustrating an example configuration of an acoustic reproduction system.

[0017] Figure 3 This is a diagram illustrating an example of the configuration of an information terminal device.

[0018] Figure 4 This is an example graph illustrating the sensitivity curve.

[0019] Figure 5 It is a flowchart describing the reproduction process.

[0020] Figure 6 This is an example graph illustrating the sensitivity curve.

[0021] Figure 7 This is an example graph illustrating the sensitivity curve.

[0022] Figure 8 This is an example graph illustrating the sensitivity curve.

[0023] Figure 9 This is an example graph illustrating the sensitivity curve.

[0024] Figure 10 This is an example diagram used to describe the user's motion and acoustic effects.

[0025] Figure 11 This is an example diagram used to describe the user's motion and acoustic effects.

[0026] Figure 12 This is an example diagram used to describe the user's motion and acoustic effects.

[0027] Figure 13 This is an example diagram used to describe the user's motion and acoustic effects.

[0028] Figure 14 This is a diagram illustrating an example of motion detection used to describe a user's movement.

[0029] Figure 15 This is a diagram illustrating an example of motion detection used to describe a user's movement.

[0030] Figure 16 This is an example diagram used to describe the user's motion and acoustic effects.

[0031] Figure 17 This is an example diagram used to describe the user's motion and acoustic effects.

[0032] Figure 18 This is an example diagram used to describe the user's motion and acoustic effects.

[0033] Figure 19 This is an example diagram used to describe the user's motion and acoustic effects.

[0034] Figure 20 This is an example diagram used to describe the user's motion and acoustic effects.

[0035] Figure 21 This is an example diagram used to describe the user's motion and acoustic effects.

[0036] Figure 22 This is an example diagram used to describe the user's motion and acoustic effects.

[0037] Figure 23 It is a flowchart describing the selection process.

[0038] Figure 24 This is an example of a selection screen that illustrates the sensitivity curve.

[0039] Figure 25 It is a flowchart describing the selection process.

[0040] Figure 26 This is an example graph illustrating the user's motion and sensitivity curves.

[0041] Figure 27 It is a flowchart describing the drawing process.

[0042] Figure 28 This is an example of an input screen displaying the sensitivity curve.

[0043] Figure 29 This is an example of an animation curve.

[0044] Figure 30 This is an example of an animation curve.

[0045] Figure 31 It is a flowchart describing the reproduction process.

[0046] Figure 32 This is an example of an animation curve.

[0047] Figure 33 This is an example of an animation curve.

[0048] Figure 34 It is a flowchart describing the reproduction process.

[0049] Figure 35 This is a diagram illustrating an example of a computer configuration. Detailed Implementation

[0050] In the following description, embodiments of the application of this technology will be described with reference to the accompanying drawings.

[0051] <First Embodiment>

[0052] <Configuration Example of an Acoustic Reproduction System>

[0053] When the sound changes according to the user's body movement, this technology enables the user to intuitively operate the sound by performing nonlinear acoustic processing on the acoustic signal to be reproduced based on the result of detecting the user's movement.

[0054] For example, consider the case where a DJ manipulates sound by moving their arms up and down.

[0055] In this context, in many situations, the arm moves most frequently and rapidly within the range of upward movement observed from the DJ's perspective, for example, within the range of 45 degrees or more when the arm moves upward from a forward-protruding position (horizontal position).

[0056] Therefore, if the amount of sound change increases when the DJ's arm is above and decreases when the DJ's arm is below, then the DJ should be able to manipulate the sound intuitively.

[0057] However, for example, when the output waveform of a sensor attached to the DJ's arm is directly applied to parameters and acoustic processing such as effects processing is performed on the acoustic signal based on those parameters, the sound changes linearly with respect to the position (height) of the DJ's arm, regardless of whether the DJ's arm is above or below. This creates a gap between the expected change in sound when the DJ moves their arm and the actual change in sound, making intuitive operation difficult.

[0058] Furthermore, for example, when changing the sound by performing thresholding on the position of the DJ's arm and then performing acoustic processing on the acoustic signal to be reproduced based on the result of the thresholding, the changes in the sound are discrete, making not only intuitive operation difficult, but also limiting the expression that can be made by manipulating the sound.

[0059] Therefore, in this technology, nonlinear acoustic processing is performed on the acoustic signal to be reproduced based on the user's motion.

[0060] Specifically, for example, in this technology, a function that obtains a specific curve or polyline in advance through interpolation processing is used as input to the user's motion sensing value and as output the sensitivity corresponding to the sensing value when operating the sound, and acoustic processing is performed using parameters corresponding to the output value of the function.

[0061] In this way, the degree of change in the sound to be manipulated, that is, the sensitivity when manipulating the sound, dynamically changes according to the amplitude of the user's movement, such as the angle or position of the user's body parts, or the speed or intensity of the movement, and the user can perform intuitive operations on the sound. In other words, the user can easily reflect their intentions when manipulating the sound.

[0062] The technology will be described in more detail below.

[0063] First, the acoustic reproduction system that applies this technology will be described.

[0064] For example, such as Figure 1 As shown, the acoustic reproduction system using this technology has an instrument 11 played by the user, a wearable device 12 attached to a predetermined part of the user's body, an information terminal device 13, a speaker 14, and an audio interface 15.

[0065] In this example, for instance, musical instrument 11, information terminal device 13, and speaker 14 are connected via audio interface 15, and if the user plays musical instrument 11, the sound corresponding to the performance is reproduced by speaker 14. At this time, the reproduced performance sound changes according to the user's movement.

[0066] Note that instrument 11 can be any instrument, such as a keyboard instrument like a piano or keyboard, a string instrument like a guitar or violin, a percussion instrument like a drum, a wind instrument, or an electronic instrument like a trackpad.

[0067] In addition, the wearable device 12 is a device that can be attached to any part of the user's body (such as the arm) and includes various sensors such as accelerometers, gyroscopes, microphones, electromyography, pressure sensors, or flexion sensors.

[0068] Using sensors, the wearable device 12 detects the user's movement, more specifically, the movement of the attached parts of the user's wearable device 12, and provides the sensed values ​​indicating the detection results to the information terminal device 13 via wireless or wired communication.

[0069] Note that here, an example of detecting a user's movement via wearable device 12 will be described. However, it is not limited to this; a user's movement can be detected by sensors (such as cameras or infrared sensors) arranged around the user in a state not attached to the user, or such sensors can be set on musical instrument 11.

[0070] Furthermore, such sensors and wearable devices 12 arranged around the user can be combined to detect the user's movements.

[0071] Information terminal device 13 is, for example, a signal processing device such as a smartphone or tablet computer. Note that it is not limited to this; information terminal device 13 can be any signal processing device such as a personal computer.

[0072] exist Figure 1 In the acoustic reproduction system shown, for example, when playing an instrument 11 while the wearable device 12 is attached to the user, the user performs desired movements (actions) to achieve the desired changes in sound expression based on the performance. Movements referred to herein include, for example, movements such as raising or lowering an arm or waving a hand.

[0073] Then, the acoustic signal used to reproduce the performance sound is provided from the instrument 11 to the information terminal device 13 via the audio interface 15.

[0074] Note that this description will assume that audio interface 15 is a general audio interface that inputs and outputs acoustic signals for reproducing the played sound. However, audio interface 15 could be a MIDI interface, etc., that inputs and outputs MIDI signals indicating the pitch of the played sound.

[0075] Furthermore, in the wearable device 12, the user's movements during the performance are detected, and the resulting sensing values ​​are provided to the information terminal device 13.

[0076] Then, based on the sensed values ​​provided from the wearable device 12 and a pre-prepared transformation function representing the sensitivity curve, the information terminal device 13 calculates the acoustic parameters to be performed on the acoustic signal. These acoustic parameters vary non-linearly with respect to the sensed values.

[0077] The information terminal device 13 performs acoustic processing on the acoustic signal provided from the musical instrument 11 via the audio interface 15 based on the obtained acoustic parameters, and provides the resulting reproduced signal to the speaker 14 via the audio interface 15.

[0078] The speaker 14 outputs sound based on the reproduced signal provided from the information terminal device 13 via the audio interface 15. With this arrangement, the sound of the playing instrument 11 is reproduced with acoustic effects such as effects corresponding to the user's movements.

[0079] Here, the sensitivity curve is a non-linear curve or broken line indicating the sensitivity characteristics when the user performs an operation on the played sound (that is, the addition of acoustic effects) through movement, and the function representing the sensitivity curve is a transformation function.

[0080] In this example, for instance, the sensed values ​​of the detected results indicating the user's motion are substituted into the transformation function, and the calculation is performed.

[0081] Then, as a result of the calculation, namely the output value of the transformation function (hereinafter referred to as the function output value), a value indicating the intensity (amplitude) of the acoustic effect added to the user's movement is obtained (that is, sensitivity).

[0082] Furthermore, in the information terminal device 13, acoustic parameters are calculated based on the function output value, and acoustic processing to add acoustic effects is performed based on the obtained acoustic parameters.

[0083] For example, acoustic effects added to an acoustic signal include various effects such as delay, pitch bend, translation, or volume changes caused by gain correction.

[0084] Therefore, for example, when adding pitch bend as an acoustic effect, the acoustic parameter is a value that indicates the amount of pitch displacement when pitch bends.

[0085] In acoustic processing, nonlinear acoustic processing can be achieved by using acoustic parameters obtained from the output value of a transformation function representing a nonlinear sensitivity curve. In other words, the sensitivity can be dynamically changed based on the user's body movements.

[0086] This arrangement allows for a full reflection of the user's intent, and allows the user to perform intuitive operations on the sound while playing instruments such as instrument 11, that is, to add acoustic effects.

[0087] Note that transformation functions can be prepared in advance, or users can create desired motions and add transformation functions that correspond to new acoustic effects.

[0088] In such cases, for example, the information terminal device 13 can download a pre-prepared desired transformation function from a server via a wired or wireless network, or upload something obtained by associating a user-created transformation function with information indicating motion to a server.

[0089] Additionally, for example, an acoustic reproduction system using this technology can have... Figure 2 The configuration shown is as follows. Note that in... Figure 2 In, with Figure 1 Corresponding parts in the figures are given the same reference numerals, and descriptions of the corresponding parts will be omitted as appropriate.

[0090] exist Figure 2 In the example shown, musical instrument 11 and information terminal device 13 are connected wirelessly or via a wired connection such as an audio interface or MIDI interface, and information terminal device 13 and wearable device 12 are connected wirelessly or via a wired connection.

[0091] In this scenario, for example, information terminal device 13 receives an acoustic signal from musical instrument 11, performs acoustic processing on the acoustic signal based on acoustic parameters obtained from sensing values ​​provided by wearable device 12, and generates a reproduced signal. Then, information terminal device 13 reproduces the sound based on the generated reproduced signal.

[0092] Additionally, sound can be reproduced on the instrument 11 side. In such cases, for example, the information terminal device 13 can provide the instrument 11 with a MIDI signal corresponding to the reproduced signal to reproduce the sound, or the information terminal device 13 can send sensing values, acoustic parameters, etc. to the instrument 11, and perform acoustic processing on the instrument 11 side.

[0093] Note that in the following description, it will be assumed that the information terminal device 13 receives an acoustic signal provided from the musical instrument 11 and reproduces the sound in the information terminal device 13 based on the reproduced signal.

[0094] <Configuration Example of Information Terminal Equipment>

[0095] Next, the description Figure 1 and Figure 2 The configuration example of the information terminal device 13 shown is shown.

[0096] For example, information terminal equipment 13, such as Figure 3 The configuration shown is as shown.

[0097] Figure 3 The information terminal device 13 shown has a data acquisition unit 21, a sensing value acquisition unit 22, a control unit 23, an input unit 24, a display unit 25, and a speaker 26.

[0098] The data acquisition unit 21 is connected to the instrument 11 via wired or wireless means to acquire the acoustic signals output from the instrument 11 and provide the acoustic signals to the control unit 23.

[0099] Note that although the case described here is the sound of playing instrument 11, which is the acoustic signal to be reproduced, it is not limited to this. The acoustic signal of any sound can be acquired by the data acquisition unit 21 as the object of reproduction.

[0100] Therefore, for example, when an acoustic signal such as pre-recorded music is acquired by the data acquisition unit 21, acoustic processing is performed to add acoustic effects to the acoustic signal, and the music with added acoustic effects is reproduced.

[0101] Additionally, for example, the acoustic signal to be reproduced can be the sound signal of an acoustic effect, i.e., the sound effect (effect sound) itself, and the intensity of the effect in the sound effect can change according to the user's movement. Furthermore, the sound effect whose intensity changes according to the user's movement can be reproduced along with the playing sound of instrument 11.

[0102] The sensing value acquisition unit 22 is connected to the wearable device 12 via wired or wireless means, acquires sensing values ​​from the wearable device 12 indicating the movement of the attached part of the wearable device 12 on the user, and provides the sensing values ​​to the control unit 23.

[0103] Note that the sensing value acquisition unit 22 can acquire sensing values ​​that indicate the movement of the instrument (in other words, the movement of the user operating the instrument) from sensors installed on an instrument such as a musical instrument 11 played by a user.

[0104] The control unit 23 controls the operation of the entire information terminal device 13. In addition, the control unit 23 has a parameter calculation unit 31.

[0105] The parameter calculation unit 31 calculates acoustic parameters based on the sensing values ​​provided by the sensing value acquisition unit 22 and the pre-saved transformation function.

[0106] The control unit 23 performs nonlinear acoustic processing on the acoustic signal provided by the data acquisition unit 21 based on the acoustic parameters calculated by the parameter calculation unit 31, and provides the resulting reproduced signal to the loudspeaker 26.

[0107] The input unit 24 includes, for example, a touch panel, buttons, switches, etc. superimposed on the display unit 25, and provides signals corresponding to the user's operation to the control unit 23.

[0108] The display unit 25 includes, for example, a liquid crystal display panel, and displays various images under the control of the control unit 23. The speaker 26 reproduces sound based on the reproduction signal provided from the control unit 23.

[0109] <Regarding the sensitivity curve>

[0110] Here, we will describe the transformation function used to calculate the acoustic parameters, that is, the sensitivity curve represented by the transformation function.

[0111] For example, the sensitivity curve is a non-linear curve, etc. Figure 4 As shown. Note that in Figure 4 In the diagram, the horizontal axis represents the user's motion, i.e., the sensed value, while the vertical axis represents the sensitivity, i.e., the function output value.

[0112] In particular, Figure 4In the example shown, the sensitivity to changes in the sensed value varies greatly between the range of small sensed value and the range of large sensed value, and the transformation function is a nonlinear function.

[0113] Furthermore, in this example, the function output value obtained by substituting the sensed value into the transformation function is set to a value between 0 and 1.

[0114] Such a sensitivity curve can be obtained, for example, by specifying two or more combinations of predetermined points (i.e., sensed values) and the corresponding sensitivities (function output values), and performing interpolation based on the specified points and a specific Bézier curve. In other words, interpolation is performed between two or more points determined for the specified points based on the Bézier curve, and the sensitivity curve is obtained.

[0115] Therefore, when using a transformation function representing such a sensitivity curve, the acoustic parameters change non-linearly along the sensitivity curve. In other words, the amount of change in the sound produced by instrument 11 can dynamically change along the sensitivity curve based on the user's movement.

[0116] For example, within a range of values ​​that can be used as sensing values, sensitivity can be seamlessly changed by connecting a range where the sensitivity to changes in sound in response to a user's movement is expected to be low and a range where the sensitivity is expected to be high.

[0117] Furthermore, using sensitivity curves can expand the user's range of musical expression because, unlike discrete changes to sound through thresholding, sound can be changed non-linearly and continuously.

[0118] <Instructions for Reproduction Processing>

[0119] Next, the operation of information terminal device 13 will be explained. That is, the following text will refer to... Figure 5 The flowchart in the diagram illustrates the reproduction process of the information terminal device 13.

[0120] Reproduction processing begins when a user with the attached wearable device 12 plays the instrument 11 while performing the desired movement.

[0121] In step S11, the data acquisition unit 21 acquires the acoustic signal output from the instrument 11 and provides the acoustic signal to the control unit 23.

[0122] In step S12, the sensing value acquisition unit 22 receives sensing values ​​from the wearable device 12 via wireless communication or the like to acquire sensing values ​​indicating the user's motion and provides the sensing values ​​to the control unit 23.

[0123] In step S13, the parameter calculation unit 31 substitutes the sensing value provided by the sensing value acquisition unit 22 into the pre-saved transformation function and performs calculations to obtain the function output value.

[0124] Note that for multiple user movements, parameter calculation unit 31 can maintain the transformation function corresponding to each movement, and can use the transformation function corresponding to the movement indicated by the sensing value in step S13.

[0125] Alternatively, for example, the function output value can be obtained by using a transformation function selected by the user or other pre-operated input unit 24 from a plurality of pre-saved transformation functions.

[0126] In step S14, the parameter calculation unit 31 calculates the acoustic parameters based on the function output value obtained in step S13.

[0127] For example, parameter calculation unit 31 calculates acoustic parameters by performing a scale transformation on the function output value to a scale for the acoustic parameters. Therefore, the acoustic parameters change non-linearly based on the sensed values.

[0128] In this case, since the function output value can be said to be a normalized acoustic parameter, the transformation function can be said to be a function that takes the user's motion (motion amount) as input and the amount of sound change (i.e. acoustic parameter) caused by the acoustic effect as output.

[0129] In step S15, the control unit 23 generates a reproducible signal by performing nonlinear acoustic processing on the acoustic signal acquired in step S11 and provided by the data acquisition unit 21 based on the acoustic parameters obtained in step S14.

[0130] In step S16, the control unit 23 provides the reproduction signal obtained in step S15 to the speaker 26 to reproduce the sound, and the reproduction process ends.

[0131] By outputting sound based on the reproduced signal in the speaker 26, the playing sound of the instrument 11, with acoustic effects added according to the user's motion, is reproduced.

[0132] As described above, the information terminal device 13 calculates acoustic parameters based on the sensed values ​​and a transformation function representing a nonlinear sensitivity curve, and performs nonlinear acoustic processing on the acoustic signal based on these acoustic parameters. In this way, the sensitivity of sound operation can be dynamically changed, and the user can operate the sound intuitively.

[0133] <Another example of a sensitivity curve>

[0134] Note that the sensitivity curve represented by the transformation function is not limited to... Figure 4The example shown can be any other sensitivity curve, as long as it is a non-linear curve or a piecewise linear curve.

[0135] For example, the sensitivity curve can be as follows: Figure 6 The exponential function curve is shown below. Note that in... Figure 6 In the diagram, the horizontal axis represents the user's body movement, i.e., the sensing value, while the vertical axis represents the sensitivity, i.e., the function output value.

[0136] For example, similar to Figure 4 The example shown, Figure 6 The sensitivity curve shown can be obtained by interpolation based on a Bézier curve, and in this example, the transformation function representing the sensitivity curve is an exponential function.

[0137] In such a sensitivity curve, the sensitivity, i.e. the function output value, decreases as the user's movement decreases, and conversely, the function output value increases as the user's movement increases.

[0138] Furthermore, the motion of the user's body input to the transformation function, i.e., the sensed value, can be, for example, the user's acceleration in the direction of each of the x, y, and z axes in the three-dimensional xyz space, the combined acceleration of these accelerations, the jerk of the user's motion, the user's rotation angle (tilt angle) with each of the x, y, and z axes as the rotation axis, etc.

[0139] In addition, the sensed values ​​can be the sound pressure level or each frequency component of the aerodynamic sound generated by the user's movement, the main frequency of the aerodynamic sound, the distance the user moves, the muscle contraction state measured by an electromyography instrument, the pressure of the user pressing the keyboard, etc.

[0140] The nonlinear transformation function of the sensitivity curve can be obtained by appropriately using a curve such as a Bezier curve to perform interpolation, so that the sensitivity varies nonlinearly according to the magnitude of the sensed value of the user's motion, such as rotation or movement, obtained in this way.

[0141] In addition, such as Figure 7 and Figure 8 The curve shown can be used as a sensitivity curve obtained by interpolation based on the Bézier curve.

[0142] Please note that in Figure 7 and Figure 8 In the graph, each curve represents a sensitivity curve, and the name of the curve is written at the bottom of each sensitivity curve graph. Furthermore, in each sensitivity curve, the horizontal direction (horizontal axis) indicates the user's movement, while the vertical direction (vertical axis) indicates the sensitivity.

[0143] By utilizing Figure 7 and Figure 8 The sensitivity curve (transformation function) shown indicates that the amount of change in the played sound can be changed curvilinearly (non-linearly) according to the user's movement.

[0144] In particular, even Figure 7 and Figure 8 Some of the sensitivity curves shown have similar shapes, and the way the sensitivity changes also depends on the angle of the curved portion on each sensitivity curve.

[0145] For example, when a curve of type easeIn, which includes "easeIn" in its name, is used as a sensitivity curve, the change in sound decreases as the user's body movement decreases, and the change in sound increases as the user's body movement increases.

[0146] Conversely, for example, when using a curve of type EaseOut, which includes "EaseOut" in its name, the change in sound increases as the user's body movement decreases, and the change in sound decreases as the user's body movement increases.

[0147] As mentioned above, depending on the angle or starting position of the curved section, even curves with similar shapes may have different locations or amounts of sensitivity variation.

[0148] Furthermore, when using a type of curve called easeInOut, the amount of sound change is small within a small range of user body movement, increases rapidly when the user body movement is moderate, and is small within a large range of user body movement.

[0149] When using a curve of the type called Elastic, sound can be expressed as if it expands or contracts with changes in the user's body movement. When using a curve of the type called Bounse, sound can be expressed as if it bounces (jumps) with changes in the user's body movement.

[0150] In addition to curves obtained through interpolation using Bézier curves, for example, any non-linear curve or polyline (such as...) can be used. Figure 9 The broken line or curve shown in the figure serves as a sensitivity curve.

[0151] Please note that in Figure 9 In the diagram, the horizontal axis represents the user's motion, i.e., the sensed value, while the vertical axis represents the sensitivity, i.e., the function output value.

[0152] For example, in the example shown by arrow Q11, the sensitivity curve is a broken line with a triangular waveform, while in the example shown by arrow Q12, the sensitivity curve is a broken line with a rectangular waveform. Furthermore, in the example shown by arrow Q13, the sensitivity curve is a periodic sine curve.

[0153] <Examples of motion and acoustic effects>

[0154] In addition, specific examples of the user's motion and the acoustic effects added based on that motion will be described.

[0155] For example, such as Figure 10 As shown, when the DJ, acting as the user, moves the user's hand (arm) in the vertical direction, i.e., the direction indicated by arrow W11, the sound based on acoustic signals can be altered. For example, the angle of the user's arm movement can be detected (measured) using a gyroscope sensor or the like provided in the wearable device 12.

[0156] In this context, for example, the acoustic effect to be applied to an acoustic signal could be a delay effect known as an echo effect, achieved by a delay filter, or a filter effect achieved by cutting off low frequencies using a cutoff filter.

[0157] In this case, the control unit 23 performs filtering processing of the delay filter or the cutoff filter as a nonlinear acoustic process.

[0158] In particular, in this case, if the representation is used Figure 7 and Figure 8 The transformation function of the easyIn sensitivity curve shown in the diagram, along with changes in sound delay (i.e., the degree of acoustic effect application), decreases as the angle of the user's arm movement decreases (i.e., the arm angle becomes closer to a horizontal position). In other words, the so-called dry component increases, while the wet component decreases. Conversely, as the angle of the user's arm increases, the change in sound increases.

[0159] Conversely, the change in sound may increase as the angle of the user's arm decreases, and the change in sound may decrease as the angle of the user's arm increases.

[0160] In addition, for example, such as Figure 11 As shown, when the DJ, acting as the user, moves the user's hand (arm) in the lateral direction, i.e., in the direction indicated by arrow W21, the sound based on the acoustic signal can be altered.

[0161] At this point, for example, depending on the user's arm position in the lateral direction, effects such as shifting the sound image position based on the lateral translation of the acoustic signal can be added as acoustic effects. Specifically, in this case, as the angle of the user's arm in the lateral direction increases, it is conceivable to shift the sound source (sound) to a greater extent, i.e., to move the sound image position to a greater extent. Conversely, as the angle of the user's arm in the lateral direction decreases, the sound can be shifted to a greater extent.

[0162] In addition, for example, such as Figure 12 As shown, when a user performs a rapid motion with their finger, effects such as reverberation, distortion, or pitch bend (i.e., acoustic effects) can be added to the acoustic signal.

[0163] In this case, the user's rapid movements can be detected by sensing the vibration (i.e., jolt) applied to the wearable device 12 attached to the user's wrist or the like during the rapid movements.

[0164] Then, in the information terminal device 13, acoustic processing such as filtering to add effects is performed based on the sensed value of the jolt, so that the amount of change of effects (acoustic effects) such as reverberation is changed.

[0165] Additionally, for example, such as Figure 13 As shown, when a user performs the action of shaking their fingers or arm in the horizontal direction, i.e., the direction indicated by arrow W31, as a movement when playing a keyboard instrument such as a piano as instrument 11, effects such as pitch bend or vibrato (acoustic effects) can be added.

[0166] In this case, for example, the action of shaking the arm in the lateral direction is detected by an accelerometer or the like provided in a wearable device 12 attached to the user's wrist, and an acoustic effect is added based on the acceleration value obtained as a sensing value as a detection result.

[0167] Specifically, for example, as the acceleration value, which is the sensed value, increases, the pitch displacement in pitch bend, which is the acoustic effect, can increase; and conversely, as the acceleration value decreases, the pitch displacement can decrease. In this example, the pitch displacement in pitch bend is considered as an acoustic parameter.

[0168] In addition, Figure 13 In the examples, for instance, it can be done through, as... Figure 14 The pressure sensors shown are installed in each keyboard section (such as the keyboard KY11 section) of the piano, which is an instrument 11, to detect the swaying of the user's arm (fingers) in the lateral direction as a movement.

[0169] In this case, based on the output values ​​of the pressure sensors set in each keyboard section, it is possible to identify which key is pressed at each time (timed), and based on the identified results, it is possible to detect the lateral movement of the user's arm.

[0170] In addition, Figure 13 In the examples, for instance, it can be done through, as... Figure 15 The sensor CA11 (such as a camera or infrared sensor) shown is placed on the part in front of the user, which is the piano, as an instrument 11, to detect the swaying of the user's arm (fingers) in the lateral direction as a movement.

[0171] For example, if a user's motion is detected by a camera that acts as a sensor CA11, the amplitude of the user's sway in the lateral direction is obtained from the motion image captured by the camera on the instrument 11 side or in the sensing value acquisition unit 22, and the value indicating the amplitude of the sway is used as a sensing value.

[0172] In addition, for example, such as Figure 16 As shown, when a user performs the action of shaking their arm in the vertical direction, i.e., the direction indicated by arrow W41, as a movement when playing a keyboard instrument such as a piano as instrument 11, an acoustic effect can be added.

[0173] In this context, for example, based on the amplitude of the user's arm movement (shaking) in the vertical direction, variations in volume levels or effects such as drive, distortion, or resonance can be added as acoustic effects to the performance sound based on the acoustic signal. In this case, the amount of sound change, i.e., the intensity of the added acoustic effect, also varies according to the detected amplitude of the sway.

[0174] In addition, for example, such as Figure 17 As shown, acoustic effects can be added when playing a keyboard instrument such as a piano as instrument 11, and the user performs an arm-swinging motion to the left or right as indicated by arrows W51 or W52, as a motion when pressing the keyboard with fingers.

[0175] In this case, pitch bend is added as an acoustic effect, and for example, as the user moves his arm to the right as shown by arrow W51, the playing sound of instrument 11 is shifted to a higher note by pitch bend; and conversely, as the user moves his arm to the left as shown by arrow W52, the playing sound is shifted to a lower note by pitch bend.

[0176] In addition, for example, such as Figure 18As shown, acoustic effects can be added when playing a keyboard instrument such as a piano as instrument 11, and the user performs arm movements to the right and left as indicated by arrow W61, as if pressing the keyboard with their fingers.

[0177] In this case, the user's arm rotation angle to the right and left is detected as a sensing value, and effects such as pitch bend are added to the playing sound as an acoustic effect based on the rotation angle.

[0178] In addition, for example, such as Figure 19 As shown, when a user performs actions such as shaking the guitar strings or moving their head (neck) as when playing a stringed instrument such as a guitar as instrument 11, acoustic effects such as vibrato or pitch bend can be added.

[0179] In this case, for example, when a user shakes their hand or fingers while pressing the strings as shown by arrow W71, or shakes their head up and down as shown by arrow W72, the sound of playing a guitar or the like is performed as an acoustic effect by adding vibrato or bend.

[0180] In this case, for example, the sensing value acquisition unit 22 can acquire sensing values ​​indicating the movement of the head part of the guitar or the like from a sensor provided on the guitar or the like, which is a musical instrument 11, or it can acquire sensing values ​​output from the wearable device 12 as sensing values ​​indicating the movement of the head part.

[0181] In addition, for example, such as Figure 20 As shown, the user's action of pressing the board (keyboard) of the trackpad of instrument 11 or the keyboard of a keyboard instrument (such as a piano), in particular the force (pressure) of pressing the board or keyboard, can be detected as motion, and acoustic effects can be added based on the detected pressure.

[0182] In this case, instead of the wearable device 12, a pressure sensor located on the keypad (keyboard) portion of the instrument 11 detects the user's movement (the force with which they press the keypad, etc.). Therefore, for example, if the user shakes their hand while pressing the keypad, the pressure applied to the keypad changes according to the shaking, and thus the intensity of the added acoustic effect also changes.

[0183] Similarly, for example, such as Figure 21 As shown, the striking force (pressure) of a percussion instrument (such as a drum) that is instrument 11 can be detected by a pressure sensor or the like installed on the percussion instrument, and effects (acoustic effects) can be added to the playing sound of the drum or the like based on the detection results.

[0184] In this case, the sound of a drum or similar instrument is collected by a microphone, and the resulting acoustic signal can be acquired by the data acquisition unit 21. In this way, the control unit 23 can perform nonlinear acoustic processing on the acoustic signal of the drum or similar instrument's sound based on acoustic parameters. Note that even without collecting the sound of a drum or similar instrument's sound, an acoustic sound effect with an intensity corresponding to the acoustic parameters can be reproduced from the speaker 26 along with the sound of the instrument's performance.

[0185] In addition, for example, such as Figure 22 As shown, the user's tilting motion of the wind instrument 11 in the direction indicated by arrow W81 can be detected as motion, and depending on the degree of tilt, acoustic effects can be added to the acoustic signal of the sound played by the instrument 11. In this case, the sound of the wind instrument can be obtained by collecting sound using a microphone. Furthermore, not only for wind instruments but also for string instruments such as guitars, the motion of tilting a string instrument can be detected as motion.

[0186] <Regarding the selection of sensitivity curves>

[0187] Furthermore, if multiple sensitivity curves, i.e. multiple transformation functions, are prepared before the acoustic parameters are calculated in the parameter calculation unit 31, the desired sensitivity curve can be selected from the sensitivity curves and used for the calculation of acoustic parameters.

[0188] For example, when multiple sensitivity curves are prepared in advance, methods such as using the default preset sensitivity curve, the user selecting a sensitivity curve from multiple sensitivity curves, and using a sensitivity curve corresponding to the type of motion are considered.

[0189] For example, if a sensitivity curve is preset for motion by default, when a user performs a specific motion, the parameter calculation unit 31 receives the sensing value corresponding to that motion from the sensing value acquisition unit 22.

[0190] Then, the parameter calculation unit 31 calculates acoustic parameters based on a transformation function representing a preset sensitivity curve for the motion performed by the user, and based on the provided sensing values.

[0191] Therefore, in this case, if the user makes a specific motion, the sound of the instrument 11 will automatically change along a preset sensitivity curve from the user's point of view.

[0192] Specifically, for example, when a user performs a shaking arm movement, suppose a transformation function representing an exponential curve is preset for the sensing value indicating the arm shaking. In this case, the sensitivity is low when the arm shaking is small, and automatically increases as the arm shaking becomes larger, and the change in sound also increases.

[0193] <Explanation of Processing Selection>

[0194] Furthermore, when the user selects a sensitivity curve from multiple sensitivity curves, for example, the selection process for selecting a sensitivity curve based on the user's instructions is executed at a timed interval provided by the user.

[0195] In the following text, reference will be made to Figure 23 The flowchart in the diagram illustrates the selection process performed by the information terminal device 13.

[0196] In step S41, by reading image data from a memory (not shown) and providing the image data to the display unit 25, the control unit 23 displays the image data as a selection screen for the graphical user interface (GUI).

[0197] Using this arrangement, for example, Figure 24 The selection screen for the sensitivity curve (transformation function) shown is displayed on the display unit 25.

[0198] exist Figure 24 In the example shown, the selection screen is displayed on the display unit 25, and multiple sensitivity curves and their names that are pre-saved in the parameter calculation unit 31 are displayed as a list on the selection screen.

[0199] Users specify (select) the desired sensitivity curve from a list of multiple sensitivity curves displayed by touching the curve with their fingers or other means.

[0200] In this example, the touch panel, serving as input unit 24, is superimposed on display unit 25, and when the user performs a touch operation in the area displaying the sensitivity curve, a signal corresponding to the touch operation is provided from input unit 24 to control unit 23. Note that the user may be able to select the sensitivity curve for each motion.

[0201] Return to Figure 23 As described in the flowchart, in step S42, the control unit 23 selects a transformation function representing the sensitivity curve specified by the user from among multiple sensitivity curves displayed on the selection screen based on the signal provided from the input unit 24, as the transformation function to be used to calculate the acoustic parameters.

[0202] When the sensitivity curve, i.e., the transformation function, is selected in this way, it will be executed subsequently. Figure 5In step S13 of the reproduction process, by using in Figure 23 In step S42, the transformation function is selected to obtain the function output value.

[0203] The selection process ends when the transformation function is selected by the control unit 23 and the information indicating the selection result is recorded by the parameter calculation unit 31 of the control unit 23.

[0204] As described above, the information terminal device 13 displays a selection screen and allows the user to select a transformation function according to their instructions. In this way, not only can the transformation function be switched according to the user's preferences or desired applications, but acoustic effects can also be added along the user's desired sensitivity curve.

[0205] <Explanation of Processing Selection>

[0206] Furthermore, when selecting a sensitivity curve from multiple sensitivity curves that corresponds to the type of motion, i.e., when the sensitivity curve changes according to the user's motion, execution is performed. Figure 25 The selection process shown is used as a selection process.

[0207] In the following text, reference will be made to Figure 25 The flowchart in the diagram illustrates the selection process performed by the information terminal device 13. Note that the following is a reference: Figure 25 The description of the selection process in the reference Figure 5 The reproduction process described begins in step S12 when the sensed value is acquired.

[0208] In step S71, the parameter calculation unit 31 identifies the type of user's motion based on the sensing values ​​provided by the sensing value acquisition unit 22.

[0209] For example, the type of motion can be identified based on the following: the time variation of the sensed value, information provided along with the sensed value from the wearable device 12 and indicating the type of sensor that has been used to obtain the sensed value, etc.

[0210] In step S72, the parameter calculation unit 31 selects the transformation function for the sensitivity curve determined by the motion type identified in step S71 from the transformation functions of multiple pre-saved sensitivity curves, and selects the end of processing.

[0211] After selecting the transform function for the sensitivity curve in this manner, Figure 5 In step S13 of the reproduction process, the function output value is obtained by using the transformation function selected in step S72.

[0212] Note that the type of motion that leads to the selection of which sensitivity curve transformation function can be predetermined or can be specified by the user.

[0213] As described above, the information terminal device 13 identifies the user's motion type based on sensing values, and selects a sensitivity curve (transformation function) based on the identification result. In this way, acoustic effects with appropriate sensitivity can be added to each type of motion.

[0214] For example, as by Figure 26 As indicated by arrow Q31, it is assumed that the user is performing a lateral hand-shaking motion while playing the piano as instrument 11.

[0215] In this case, for example, in parameter calculation unit 31, in step S72, a transformation function of a curve called "easeInExponential" is selected as the sensitivity curve. In other words, the easeInExponential function is selected as the transformation function.

[0216] Suppose that from this state, the user stops the lateral shaking motion of the hand that is playing, and, for example, as indicated by arrow Q32, the user performs a motion that tilts the hand that is playing the piano as instrument 11.

[0217] Then, in Figure 25 In step S72, during the newly executed selection process, a transformation function called "easeOutExpo" is selected as the sensitivity curve. In other words, the easeOutExponential function is selected as the transformation function.

[0218] Using this arrangement, the transformation function switches from the easeInExponential function to the easeOutExponential function based on changes in the user's motion type.

[0219] exist Figure 26 In the example shown, when the user shakes their hand, the sensitivity is low and the change in sound is small with small shaking, and the sensitivity gradually increases and the change in sound becomes larger as the shaking of the hand becomes larger.

[0220] Conversely, when a user performs a motion that tilts their hand, the sensitivity is high even with a small tilt, and the sound changes significantly. However, with a large tilt, the sensitivity gradually decreases, and the sound changes become more subtle.

[0221] Note that although an example of selecting a sensitivity curve based on the user's type of movement is described here, sensitivity curves or acoustic effects can also be selected based on the type of instrument 11, the type of music (genre), etc.

[0222] For example, the type of instrument 11 can be identified by the control unit 23 connecting to the instrument 11 via the data acquisition unit 21 and acquiring information indicating the type of the instrument 11 from the instrument 11. Furthermore, for example, the control unit 23 can identify the type of instrument 11 by recognizing the user's movements while playing the instrument 11 based on sensing values ​​provided from the sensing value acquisition unit 22.

[0223] Furthermore, for example, the type (genre) of the music based on the sound of the acoustic signal to be reproduced can be identified by the control unit 23 performing various analysis processes on the acoustic signal provided from the data acquisition unit 21, or it can be identified from the metadata of the acoustic signal.

[0224] <Explanation of Processing Selection>

[0225] In addition to selecting the desired sensitivity curve from a number of pre-prepared sensitivity curves, users can also specify the desired sensitivity curve by inputting a sensitivity curve through methods such as plotting a sensitivity curve.

[0226] In this situation, execution is performed in information terminal device 13. Figure 27 The drawing process is shown below. (Refer to the following text.) Figure 27 The flowchart in the document describes the drawing process of the information terminal device 13.

[0227] In step S101, the control unit 23 controls the display unit 25 to display a sensitivity curve input screen for inputting sensitivity curves on the display unit 25.

[0228] Using this arrangement, for example, Figure 28 The sensitivity curve input screen shown is displayed on the display unit 25.

[0229] exist Figure 28 In the example shown, a user can specify any sensitivity curve by drawing a sensitivity curve on the sensitivity curve input screen using a finger or similar tool, where the horizontal axis represents motion and the vertical axis represents sensitivity.

[0230] In this example, the touch panel, which serves as the input unit 24, is superimposed on the display unit 25, and the user inputs the desired sensitivity curve, such as a non-linear curve or a broken line, by performing a drawing operation on the sensitivity curve input screen using a finger or the like.

[0231] Note that the method for inputting the sensitivity curve is not limited to this, and any method can be used. Furthermore, for example, a preset sensitivity curve can be displayed on the sensitivity curve input screen, and the user can input the desired sensitivity curve by deforming the sensitivity curve through touch operations or other means.

[0232] Return to Figure 27 As explained in the flowchart, in step S102, based on the signal provided from the input unit 24 according to the user's operation of drawing the sensitivity curve, the parameter calculation unit 31 generates and records a transformation function representing the sensitivity curve input by the user. The drawing process ends when the transformation function of the sensitivity curve drawn by the user is recorded.

[0233] As described above, the information terminal device 13 generates and records a transformation function representing a sensitivity curve freely drawn by the user.

[0234] Using this setup, users can specify their desired sensitivity curve by finely adjusting or customizing the sensitivity when operating the sound based on their movements, and further, they can operate the sound intuitively.

[0235] <Second Embodiment>

[0236] <Regarding the addition of animation effects>

[0237] Incidentally, an example of adding acoustic effects to the playing sound of the instrument 11 with a sensitivity corresponding to the user's movement has been described above.

[0238] However, this is not the only possibility. For example, when a user performs a specific motion, animated effects can be added as acoustic effects to the sound to be reproduced based on the type of motion for a certain period of time. Note that in the following text, the user's specific motion is also specifically referred to as a pose.

[0239] Here, for example, an animation effect is an acoustic effect in which effects are added to the sound to be reproduced along an animation curve obtained by interpolation based on a Bézier curve over a certain period of time.

[0240] For example, animation curves can be Figure 29 The curve shown. Note that in... Figure 29 In the diagram, the vertical axis represents sound changes, while the horizontal axis represents time.

[0241] For example, in the case of an animation effect where the volume level changes over time, the change in sound indicated by the value on the vertical axis of the animation curve can be said to represent the volume level.

[0242] In the following text, the function representing the animation curve is called the animation function. Therefore, the value on the vertical axis of the animation curve, that is, the value indicating the change in sound, is the output value of the animation function (hereinafter referred to as the function output value).

[0243] For example, suppose the animation effect is the effect of changing the volume level of the sound to be reproduced, when along... Figure 29 The animation curve shown in the figure adds animation effects to the sound to be reproduced, and the volume level of the sound to be reproduced decreases over time.

[0244] This section will describe the poses and specific examples of animation effects when adding them.

[0245] For example, in the sensing value acquisition unit 22, the swinging of the user's arm in the horizontal or vertical direction can be detected as a posture based on the sensing value, and when the posture is detected, the sound of a sound source predetermined for the posture (more specifically, the type of posture) can be reproduced (hereinafter also referred to as posture sound).

[0246] At this point, add the following animation effect: for example, along Figure 30 The animation curve shown depicts the volume level of the gesture sound gradually decreasing over time. Note that in... Figure 30 In the diagram, the vertical axis represents the sound changes, i.e., the function output value of the animation function, while the horizontal axis represents time.

[0247] In this case, for example in control unit 23, animation curves and acoustic processing, i.e. animation effects, can be selected based on the detected posture.

[0248] When an animation curve is selected, the gain value, which serves as an acoustic parameter for each time period, is calculated in parameter calculation unit 31 based on the function output value at each time point. For example, the function output value is scaled to the scale of the acoustic parameter. Here, the gain value as an acoustic parameter is smaller at later times (future times).

[0249] When the acoustic parameters at each time are obtained in this way, in the control unit 23, gain correction is performed on the acoustic signal of the gesture sound based on the acoustic parameters at that time as acoustic processing, and a reproduced signal is generated.

[0250] When the sound is reproduced by the speaker 26 based on the reproduced signal obtained in this way, the reproduced sound is oriented such that the volume level decreases over time.

[0251] Furthermore, for example, actions such as pressing a keyboard or plucking a string can be detected as the movement (posture) of the user playing the instrument 11, and animation effects can be added to the playing sound of the instrument 11 along an animation curve corresponding to the user's movement within a predetermined time.

[0252] In this case, the playing sound of instrument 11 can be played as is, and the sound effects with added animation effects based on the user's movements can be reproduced together with the playing sound.

[0253] <Instructions for Reproduction Processing>

[0254] Furthermore, for example, in the sensing value acquisition unit 22, based on the sensing values ​​indicating user movement acquired at each time, the peak values ​​of the time waveforms of these sensing values ​​can be detected sequentially, and the initial values ​​of the acoustic parameters can be determined based on the detected peak values.

[0255] In such a case, for example in information terminal device 13, for example, executing Figure 31 The reproduction process is shown below. Reference will be made to... Figure 31 The flowchart in the diagram illustrates the reproduction process of the information terminal device 13.

[0256] In step S131, the sensing value acquisition unit 22 acquires sensing values ​​indicating the user's motion by receiving sensing values ​​from the wearable device 12 via wireless communication or the like.

[0257] In step S132, based on the sensing values ​​acquired so far, the sensing value acquisition unit 22 detects whether the user has performed a specific gesture.

[0258] In step S133, the sensing value acquisition unit 22 determines whether the posture has been detected as the detection result in step S132.

[0259] If it is determined in step S133 that no posture has been detected, the process returns to step S131 and the above process is repeated.

[0260] Meanwhile, if it is determined in step S133 that a posture has been detected, in step S134, the sensing value acquisition unit 22 detects the waveform peak of the sensing value based on the sensing value acquired so far within the most recent predetermined time period.

[0261] The sensing value acquisition unit 22 provides information indicating the posture and peak value detected in this manner to the parameter calculation unit 31.

[0262] In step S135, the parameter calculation unit 31 determines the animation effect (i.e., the animation curve) and acoustic processing based on the information of the detected pose and peak values ​​provided by the sensing value acquisition unit 22.

[0263] Here, for example, suppose that the animation effect and the gesture sound to be reproduced are predetermined for the type of gesture (i.e., the user's movement). In this case, the parameter calculation unit 31 selects the animation effect predetermined for the detected gesture as the animation effect to be added to the gesture sound.

[0264] Additionally, at this time, the control unit 23 controls the data acquisition unit 21 to acquire the acoustic signal of the posture sound predetermined for the detected posture.

[0265] Note that although the sound to be reproduced here is for a gesture-defined sound, it is not limited to this and animation effects can be added to any sound (such as the playing sound of instrument 11).

[0266] In step S136, the parameter calculation unit 31 calculates acoustic parameters based on information about the detected posture and peak values ​​provided by the sensing value acquisition unit 22.

[0267] In this case, for example, parameter calculation unit 31 calculates the initial value of the acoustic parameter by transforming the peak value of the sensed value into the scale of the acoustic parameter.

[0268] The initial values ​​of the acoustic parameters here are the values ​​of the acoustic parameters at the start time of the animation effect to be added to the gesture sound.

[0269] Furthermore, based on the initial values ​​of the acoustic parameters and the animation curve used to achieve the animation effect determined in step S135, the parameter calculation unit 31 calculates the acoustic parameters at various times during the time period of adding animation effects to the gesture sound.

[0270] Here, the values ​​of the acoustic parameters at each time are calculated based on the initial values ​​of the acoustic parameters and the function output values ​​of the animation function representing the animation curve at each time, so that the values ​​of the acoustic parameters gradually change along the animation curve from the initial values.

[0271] Note that in the following text, the time periods with added animation effects are also specifically referred to as animation time periods.

[0272] In step S137, the control unit 23 generates a reproduced signal by performing acoustic processing that adds animation effects to the acoustic signal of the gesture sound based on the acoustic parameters calculated in step S136 at each time.

[0273] In other words, the control unit 23 generates a reproduction signal by performing acoustic processing on the acoustic signal of the gesture sound based on acoustic parameters, while gradually changing the values ​​of the acoustic parameters along the animation curve from the initial value.

[0274] Therefore, in this case, nonlinear acoustic processing is performed on the acoustic signal because the acoustic parameters change over time.

[0275] In step S138, the control unit 23 provides the reproduction signal obtained in step S137 to the speaker 26 to reproduce the sound, and the reproduction process ends.

[0276] Using this arrangement, the gesture sound with added animation effects corresponding to the gesture is reproduced in the speaker 26.

[0277] As described above, the information terminal device 13 calculates acoustic parameters based on the peak value of the sensed value, and performs nonlinear acoustic processing on the acoustic signal based on the acoustic parameters.

[0278] In this way, users can add desired animation effects to gesture sounds simply by making predetermined gestures. Therefore, users can intuitively manipulate the sounds.

[0279] Here, we will describe a specific example of the above situation where we add animation effects that correspond to the pose.

[0280] As an example, one could imagine adding a gesture sound like this to a user's arm-swinging gesture, for instance. Figure 32 The animation curve shown is a bounce animation where the volume of the gesture sound gradually decreases.

[0281] Note that in Figure 32 In the diagram, the vertical axis represents the sound changes, i.e., the function output value of the animation function, while the horizontal axis represents time.

[0282] Figure 32 The animation curve shown is a curve where the sound gradually decreases over time as it changes up and down.

[0283] Therefore, for example, assuming that the jump when the user waves his arm is acquired as a sensing value, the peak value of the waveform of the jump as the sensing value is detected in the sensing value acquisition unit 22.

[0284] Furthermore, in parameter calculation unit 31, the gain value, which is an acoustic parameter, is determined based on the peak value of the jump, i.e., the initial value of the volume when reproducing the posture sound, and the acoustic parameters at various times are determined so that the acoustic parameters follow the curve. Figure 32 The animation curve changes shown.

[0285] Then, in the control unit 23, based on the determined acoustic parameters at each time, i.e. gain values, gain correction is performed on the acoustic signal of the gesture sound as acoustic processing, and as a result, a bounce animation effect is added to the gesture sound.

[0286] In this case, the gesture sound is reproduced due to the bounce animation effect, where the volume of the sound generated based on the user's gesture (i.e., the swing of the arm) gradually decreases over time by changing as if the sound were hitting an object and bouncing.

[0287] Additionally, one could consider adding features such as gesture sounds to them. Figure 33 The animation curve shown is an elastic animation. Note that in... Figure 33 In the diagram, the vertical axis represents the sound changes, i.e., the function output value of the animation function, while the horizontal axis represents time.

[0288] When the volume of the gesture sound along Figure 33 As the animation curve shown changes, you can add a sound generated based on the posture (posture sound) to the posture sound, making it seem as if it is elastically returning.

[0289] Furthermore, for example, the acceleration of the vibration when the percussion instrument 11 is struck can be obtained as a sensing value, and similar to the example above, various effects such as reverberation or delay can be animated by using the peak value of the sensing value indicating the vibration waveform.

[0290] In this case, the degree of application of acoustic effects such as reverberation or delay added to the playing sound of instrument 11 varies along the animation curve over time.

[0291] <First Variation of the Second Embodiment>

[0292] <Regarding the addition of animation effects>

[0293] In addition, for example, gesture sounds can be generated based on the user's movements (postures), and animation effects can be added to the gesture sounds, i.e., the waveform of the sound.

[0294] For example, suppose the acceleration indicating the user's movement is detected as a sensing value, and based on the sensing value, an acoustic signal with a sound waveform (such as a sine wave) of a specific frequency is generated as a gesture sound signal.

[0295] In such a case, it is conceivable to determine the initial values ​​of the acoustic parameters in a similar manner to the example above, and to add an animation effect to the gesture sound in which the degree of application of the effect varies over time along a predetermined animation curve.

[0296] In addition, for example, it is conceivable to add animation effects with specific waveforms to the aerodynamic sounds generated by the user's movements.

[0297] In such cases, for example, the sound pressure of aerodynamic sound is detected as a sensing value, the initial values ​​of acoustic parameters are determined based on the waveform peak value of the sensing value, and acoustic processing based on the acoustic parameters at each time is performed on the acoustic signal of aerodynamic sound obtained through sound acquisition.

[0298] <Second Variation of the Second Embodiment>

[0299] <Regarding the addition of animation effects>

[0300] Furthermore, when adding animation effects based on the user's movement, the animation effects can be added again if a large new movement of the user is detected before the animation ends.

[0301] For example, suppose the initial values ​​of acoustic parameters are determined based on the peak values ​​of the sensed values ​​that indicate the user's motion, and animation effects are added to the acoustic signal based on the initial values ​​and the animation curve to change the degree of application of the effect.

[0302] Here, although the sound based on the acoustic signal can be any sound such as the sound of the instrument 11 being played or a sound effect determined for the user's movement, it is assumed here that the sound of the instrument 11 being played is reproduced.

[0303] At this point, for example, assuming that the acceleration of a predetermined part of the user's body is detected as a sensing value, the initial values ​​of the acoustic parameters are determined based on the peak value of the acceleration.

[0304] Furthermore, when determining the initial values ​​of the acoustic parameters, the values ​​of the acoustic parameters at each subsequent time are determined so that the values ​​of the acoustic parameters vary along an animation curve determined for the user's motion, etc.

[0305] When the acoustic parameters for each time period, including the initial values, are determined in this manner, acoustic processing is performed on the acoustic signal to be reproduced based on the acoustic parameters for each time period, and a reproduced signal is generated. Then, when the sound is reproduced based on the reproduced signal obtained in this manner, animation effects for a specific time period are added to the playing sound of instrument 11 and reproduced.

[0306] In this case, if the acoustic parameter obtained for the peak of the acceleration (sensed value) indicating the user's motion exceeds the acoustic parameter at the current time before the end of the animation period, the acoustic parameter obtained for that peak is set to a new initial value.

[0307] In other words, if the acoustic parameter obtained from the peak value at any time within the animation period is greater than the actual acoustic parameter at that time, the acoustic parameter obtained from the peak value at that time is set as the initial value of the new acoustic parameter, and a new animation effect is added to the playing sound of instrument 11.

[0308] Note that while an example of adding animation effects to the playing sound of instrument 11 has been described here, similar applications can be made to other situations, such as adding animation effects to pneumatic sounds generated by the user's movements.

[0309] <Instructions for Reproduction Processing>

[0310] Here, as described above, the processing performed when the initial values ​​of the acoustic parameters are updated appropriately based on the user's movement and new animation effects are added will be described.

[0311] In other words, in the following text, reference will be made to Figure 34 The flowchart in the diagram describes the reproduction process of the information terminal device 13.

[0312] Note that here, we will describe as an example the situation where animation effects are added to the playing sound of instrument 11 when the user makes a predetermined movement.

[0313] In step S161, the data acquisition unit 21 acquires the acoustic signal output from the instrument 11 and provides the acoustic signal to the control unit 23.

[0314] In step S162, the sensing value acquisition unit 22 acquires sensing values ​​indicating the user's motion by receiving sensing values ​​from the wearable device 12 via wireless communication or the like.

[0315] In step S163, the sensing value acquisition unit 22 detects the waveform peak of the sensing value based on the sensing value acquired so far within the most recent predetermined time period.

[0316] The sensing value acquisition unit 22 provides the peak value of the sensing value detected in this way to the parameter calculation unit 31.

[0317] In step S164, the parameter calculation unit 31 calculates acoustic parameters based on the peak values ​​provided by the sensing value acquisition unit 22.

[0318] In this case, for example, parameter calculation unit 31 calculates the initial value of the acoustic parameter by transforming the peak value of the sensed value into the scale of the acoustic parameter.

[0319] In step S165, the parameter calculation unit 31 determines whether the initial value of the acoustic parameter calculated in step S164 is greater than the acoustic parameter at the current time.

[0320] For example, suppose that when a user makes a predetermined movement, a pre-determined animation effect is added to the playing sound of instrument 11 for that movement.

[0321] At this point, if the initial value of the acoustic parameter obtained in step S164 is greater than 0 when it is not an animation time period, then in step S165 it is determined that the initial value is greater than the acoustic parameter at the current time.

[0322] In addition, in the case of an animation time period, if the initial value of the acoustic parameter obtained in step S164 is greater than the acoustic parameter at the current time when the animation effect is actually added, then in step S165 it is determined that the initial value is greater than the acoustic parameter at the current time.

[0323] If it is determined in step S165 that the initial value of the acoustic parameter is not greater than the acoustic parameter at the current time, the processing in steps S166 to S168 is not performed, and the processing proceeds to step S169.

[0324] In this case, if it is not an animation period, the control unit 23 provides the speaker 26 with an acoustic signal without added acoustic effects (i.e., animation effects) as a reproduction signal, and reproduces the playing sound of the instrument 11.

[0325] Furthermore, if it is an animation segment, acoustic processing is performed on the acoustic signal based on the acoustic parameters of the current time, and the sound is reproduced by speaker 26 based on the obtained reproduced signal. In this case, the performance sound with added animation effects is reproduced.

[0326] Meanwhile, if it is determined in step S165 that the initial value of the acoustic parameter is greater than the acoustic parameter at the current time, the subsequent processing proceeds to step S166.

[0327] In this case, regardless of whether an animation effect is currently added to the playing sound of instrument 11, that is, regardless of whether it is an animation time period, the acoustic parameters of each time in the new animation time period are calculated based on the initial values ​​of the acoustic parameters calculated in step S164, and a new animation effect is added to the playing sound of instrument 11.

[0328] In step S166, the parameter calculation unit 31 calculates the acoustic parameters at each time point in the animation time period based on the initial values ​​of the acoustic parameters calculated in step S164 and the animation curve determined for the user's movement, etc.

[0329] Here, the values ​​of the acoustic parameters are calculated based on the initial values ​​of the acoustic parameters and the function output values ​​of the animation function representing the animation curve at various times, so that the values ​​of the acoustic parameters gradually change along the animation curve from the initial values.

[0330] In step S167, the control unit 23 generates a reproduced signal by performing acoustic processing that adds animation effects to the acoustic signal acquired by the data acquisition unit 21 based on the acoustic parameters calculated in step S166 for each time period.

[0331] In other words, the control unit 23 generates a reproduced signal by performing acoustic processing based on acoustic parameters on the acoustic signal and by gradually changing the values ​​of the acoustic parameters along the animation curve from their initial values.

[0332] In step S168, the control unit 23 provides the reproduction signal obtained in step S167 to the speaker 26 to reproduce the sound. With this arrangement, a new animation period begins, and animation effects are added to and reproduced from the playing sound of the instrument 11.

[0333] If the processing in step S168 is performed, or if it is determined in step S165 that the acoustic parameter is not greater than the acoustic parameter at the current time, then the control unit 23 determines in step S169 whether to end the sound reproduction based on the acoustic signal.

[0334] For example, in step S169, if the user stops playing the instrument 11, etc., it is determined that the playback will end.

[0335] If it is determined in step S169 that the reproduction has not yet ended, the process returns to step S161 and the above process is repeated.

[0336] Meanwhile, if it is determined in step S169 that the playback should be terminated, each unit of the information terminal device 13 stops the processing it is currently performing, and the playback process ends.

[0337] As described above, the information terminal device 13 calculates acoustic parameters based on the peak value of the sensed value, and performs acoustic processing on the acoustic signal based on the acoustic parameters.

[0338] In addition, when a user moves during the animation period and the value of the acoustic parameter is greater than the value of the acoustic parameter at the current time, the information terminal device 13 adds new animation effects to the playing sound of the instrument 11 based on the movement.

[0339] In this way, users can add desired animation effects based on their movements. Therefore, users can intuitively control the sound.

[0340] <Computer Configuration Example>

[0341] Incidentally, the above series of processes can be performed by hardware or by software. In the case where the processes are performed by software, the program constituting the software is installed on the computer. Here, "computer" includes computers with dedicated hardware, such as general-purpose personal computers capable of performing various functions by installing various programs.

[0342] Figure 35 This is a block diagram illustrating an example of the hardware configuration of a computer that uses a program to perform the above series of processes.

[0343] In a computer, the central processing unit (CPU) 501, read-only memory (ROM) 502, and random access memory (RAM) 503 are interconnected via bus 504.

[0344] In addition, the input / output interface 505 is connected to the bus 504. The input unit 506, output unit 507, recording unit 508, communication unit 509, and driver 510 are connected to the input / output interface 505.

[0345] Input unit 506 includes a keyboard, mouse, microphone, image sensor, etc. Output unit 507 includes a display, speaker, etc. Recording unit 508 includes a hard disk, non-volatile memory, etc. Communication unit 509 includes a network interface, etc. Driver 510 drives removable recording media 511 such as disk, optical disk, magneto-optical disk, or semiconductor memory.

[0346] In the computer configured as described above, the aforementioned series of processes are performed by the CPU 501 loading, for example, a program recorded in the recording unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executing the program.

[0347] The program executed by the computer (CPU 501) can be provided, for example, by recording on a removable recording medium 511 as an encapsulation medium. Alternatively, the program can be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0348] In a computer, a program can be installed on the recording unit 508 via the input / output interface 505 by attaching a removable recording medium 511 to the drive 510. Alternatively, the program can be received by the communication unit 509 and installed on the recording unit 508 via a wired or wireless transmission medium. Additionally, the program can be pre-installed on the ROM 502 or the recording unit 508.

[0349] Note that a program executed by a computer may be a program that is processed sequentially in the order described in this specification, or it may be a program that is processed in parallel, or it may be a program that needs to be processed at regular intervals, such as when a call is made.

[0350] Furthermore, embodiments of this technology are not limited to the above embodiments, and various changes can be made without departing from the scope of this technology.

[0351] For example, this technology can have a cloud computing configuration, in which a function is shared and processed jointly by multiple devices via a network.

[0352] Furthermore, each step described in the flowchart above can be performed by a single device or can be performed by multiple devices shared by the device.

[0353] In addition, when a step includes multiple processes, these processes can be executed by multiple devices in addition to being executed by one device.

[0354] In addition, this technology can have the following configurations.

[0355] (1) A signal processing device, comprising:

[0356] The acquisition unit acquires sensing values ​​indicating the movement of a predetermined part of a user's body or the movement of an instrument; and

[0357] A control unit performs nonlinear acoustic processing on the acoustic signal based on the sensed value.

[0358] (2) The signal processing device according to (1),

[0359] The control unit performs acoustic processing based on parameters that vary non-linearly according to the sensed values.

[0360] (3) The signal processing device according to (2),

[0361] The control unit calculates parameters corresponding to the sensed value based on a transformation function with a nonlinear curve or piecewise linear curve, wherein the transformation function is input by the user.

[0362] (4) The signal processing device according to (2),

[0363] The control unit calculates the parameters based on a transformation function selected by the user from a plurality of transformation functions used to obtain parameters from the sensed values.

[0364] (5) The signal processing device according to (2),

[0365] The control unit selects a transformation function determined by the type of motion from a plurality of transformation functions used to obtain parameters from the sensed values, and calculates the parameters based on the selected transformation function.

[0366] (6) The signal processing device according to (1),

[0367] The control unit uses acoustic processing to add animation effects to the acoustic signals.

[0368] (7) The signal processing device according to (6),

[0369] The control unit adds animation effects to the acoustic signal that are determined by the type of motion.

[0370] (8) The signal processing apparatus according to (6) or (7),

[0371] The control unit adds animation effects to the acoustic signal by obtaining initial values ​​of acoustic processing parameters based on the waveform peak value of the sensed value, and performing acoustic processing while changing the parameters from the initial values.

[0372] (9) The signal processing device according to (8),

[0373] Wherein, if at any point in the animation time period during the execution of the animation effect, the parameter corresponding to the peak value at that time is greater than the actual parameter at that time, the control unit performs acoustic processing to add new animation effects to the acoustic signal based on the initial value obtained based on the peak value at that time.

[0374] (10) The signal processing apparatus according to any one of (1) to (9),

[0375] The acoustic signal includes the sound signal of the instrument played by the user.

[0376] (11) The signal processing apparatus according to any one of (1) to (9),

[0377] Among them, acoustic signals include signals that are determined by the type of motion.

[0378] (12) A signal processing method, comprising:

[0379] From signal processing equipment:

[0380] Acquire sensor values ​​indicating the movement of a predetermined part of a user's body or the movement of an instrument; and

[0381] Nonlinear acoustic processing is performed on the acoustic signal based on the sensed value.

[0382] (13) A program that causes a computer to perform a process, the process comprising the following steps:

[0383] Acquire sensor values ​​indicating the movement of a predetermined part of a user's body or the movement of an instrument; and

[0384] Nonlinear acoustic processing is performed on the acoustic signal based on the sensed value.

[0385] Reference tag list

[0386] 11 Musical Instruments

[0387] 12 Wearable devices

[0388] 13 Information terminal equipment

[0389] 21 Data Acquisition Unit

[0390] 22 Sensing Value Acquisition Unit

[0391] 23 Control Unit

[0392] 24 Input Units

[0393] 25 display units

[0394] 26 speakers

[0395] 31 Parameter Calculation Unit

Claims

1. A signal processing device, comprising: The acquisition unit acquires sensing values ​​that indicate the movement of a predetermined part of the user's body or the movement of an instrument; as well as Control unit, the control unit: The transformation function used to obtain parameters from the sensing values ​​is selected based on the type of motion indicated by the sensing values; Use the selected transformation function to obtain the parameters corresponding to the sensed value; as well as Nonlinear acoustic processing is performed on the acoustic signal based on the obtained parameters. Specifically, the transformation function is switched from the first type of transformation function to the second type of transformation function according to the change of the type of motion.

2. The signal processing device according to claim 1, in, The control unit performs acoustic processing based on parameters that vary non-linearly according to the sensed values.

3. The signal processing device according to claim 2, in, The control unit calculates parameters corresponding to the sensed value based on a transformation function with a nonlinear curve or piecewise linear curve, the transformation function being input by the user.

4. The signal processing device according to claim 2, in, The control unit calculates the parameters based on a transformation function selected by the user from a plurality of transformation functions used to obtain parameters from the sensed values.

5. The signal processing device according to claim 1, in, The control unit uses acoustic processing to add animation effects to the acoustic signals.

6. The signal processing device according to claim 5, in, The control unit adds animation effects to the acoustic signal that are determined by the type of motion.

7. The signal processing apparatus according to claim 5, in, The control unit adds animation effects to the acoustic signal by obtaining initial values ​​of acoustic processing parameters based on the waveform peak value of the sensed value, and performing acoustic processing while changing the parameters from the initial values.

8. The signal processing apparatus according to claim 7, in, If the parameter corresponding to the peak value at any point during the animation time period is greater than the actual parameter at that time, the control unit performs acoustic processing to add new animation effects to the acoustic signal based on the initial value obtained based on the peak value at that time.

9. The signal processing apparatus according to claim 1, in, Acoustic signals include the sound signals from the playing of musical instruments by the user.

10. The signal processing apparatus according to claim 1, in, Acoustic signals include signals that are specific to the type of motion.

11. A signal processing method, comprising: From signal processing equipment: Acquire sensor values ​​indicating the movement of a predetermined part of the user's body or the movement of an instrument; The transformation function used to obtain parameters from the sensing values ​​is selected based on the type of motion indicated by the sensing values; Use the selected transformation function to obtain the parameters corresponding to the sensed value; as well as Nonlinear acoustic processing is performed on the acoustic signal based on the obtained parameters. Specifically, the transformation function is switched from the first type of transformation function to the second type of transformation function according to the change of the type of motion.

12. A program product that causes a computer to perform processing, said processing comprising the following steps: Acquire sensor values ​​indicating the movement of a predetermined part of the user's body or the movement of an instrument; The transformation function used to obtain parameters from the sensing values ​​is selected based on the type of motion indicated by the sensing values; Use the selected transformation function to obtain the parameters corresponding to the sensed value; as well as Nonlinear acoustic processing is performed on the acoustic signal based on the obtained parameters. Specifically, the transformation function is switched from the first type of transformation function to the second type of transformation function according to the change of the type of motion.

Images