Audio-visual output device and program

Abstract

To provide users with a more enjoyable and satisfying performance experience. [Solution] The sound and video output device includes: a detection unit that non-contactually detects the postures of multiple users; an instrument information acquisition unit that acquires instrument information associated with shape information indicating the shape of a figure drawn by the user's drawing operation, instrument type information indicating the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each region of the figure; a display control unit that superimposes an instrument object and an operation part image and displays them on a display screen; a coordinate calculation unit that calculates the coordinates on the display screen of the operation part image displayed on the display screen for each user based on the detected posture; an operation coordinate information and a contact area determination unit that determines the contact area for each user; and a sound output unit that outputs a sound that has a frequency assigned to the contact area and a tone corresponding to the type of instrument indicated by the instrument information for each contact area determined for each user.

Description

Technical Field

[0001] The present invention relates to an audio-visual output device and a program.

Background Art

[0002] For example, as an example of an interactive music installation, research has been conducted on an experiential installation that presents an association between visual impressions and timbres. (See, for example, Non-Patent Document 1.)

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above-described conventional experiential installation, although a method of associating visual impressions and timbres is presented, no proposal has been made regarding the configuration of a specific application. By configuring a specific application, it is preferable to provide a user with a performance experience that can give greater enjoyment and satisfaction.

[0005] An object of the present invention is to provide an audio-visual output device and a program that can give a user a performance experience that can give greater enjoyment and satisfaction.

Means for Solving the Problems

[0006] One aspect of the present invention includes a detection unit that non-contactually detects the postures of multiple users, an instrument information acquisition unit that acquires instrument information, which is generated based on the shape of a figure drawn by the user's drawing operation, and which associates shape information indicating the shape of the figure, instrument type information indicating the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each region of the figure, and an image based on the acquired instrument information, which is an instrument object that is an image of a pattern corresponding to the shape of the figure, and an image based on the detected posture, which is an image of an operating part that is at least a part of the user's body, and displays these superimposed on a display screen. The sound-video output device comprises: a display control unit; a coordinate calculation unit that calculates the coordinates on the display screen of the operation part image displayed on the display screen as operation coordinate information for each user based on the detected posture; a contact area determination unit that determines for each user, based on the operation coordinate information and the shape information of the instrument object, the contact area among a plurality of areas of the instrument object displayed on the display screen that the operation position on the display screen indicated by the operation coordinate information has come into contact with; and a sound output unit that outputs a sound for each contact area determined for each user, which has a frequency assigned to the contact area and a tone corresponding to the type of instrument indicated by the instrument information.

[0007] One aspect of the present invention involves a computer that non-contactually detects the postures of multiple users, acquires instrument information which is generated based on the shape of a figure drawn by the user's drawing operation, and which associates shape information indicating the shape of the figure, the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each region of the figure, and superimposes an image based on the acquired instrument information, which is an instrument object that is an image of a pattern corresponding to the shape of the figure, and an image based on the detected posture, which is an image of an operating part for each user that is an image indicating an operating part that is at least a part of the user's body, and displays these on a display screen. This program is designed to perform the following actions: to enable the user; to calculate the coordinates on the display screen of the image of the operating part displayed on the display screen as operation coordinate information for each user, based on the detected posture; to determine for each user, from among a plurality of areas of the instrument object displayed on the display screen, the area that the operation position on the display screen indicated by the operation coordinate information has come into contact with; and to output a sound for each contact area determined for each user, which has a frequency assigned to the contact area and a timbre corresponding to the type of instrument indicated by the instrument information. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide users with a performance experience that offers greater enjoyment and satisfaction. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the device configuration of the performance system of this embodiment. [Figure 2] This figure shows an example of the functional configuration of the instrument information generation system of this embodiment. [Figure 3] This figure shows an example of the operation flow of the instrument information generation system of this embodiment. [Figure 4] This figure shows an example of the correspondence between the drawing trajectory and the drawn figure in this embodiment. [Figure 5] This figure shows an example of the determination space for the drawn shape in this embodiment. [Figure 6] This figure shows an example of the correspondence between the shape of the drawing in this embodiment and the type of musical instrument. [Figure 7] This figure shows an example of the correspondence between the drawn shapes and musical instrument objects in this embodiment. [Figure 8] This figure shows an example of an object that can be attached to the musical instrument object of this embodiment. [Figure 9] This figure shows an example of the correspondence between the shape of the drawing figure in this embodiment and the object attached to the musical instrument object. [Figure 10] This figure shows an example of the correspondence between the area of ​​the drawn figure and the musical scale in this embodiment. [Figure 11] This figure shows an example of the correspondence between the area of ​​the drawn figure and the musical scale in this embodiment. [Figure 12] This figure shows an example of instrument information in this embodiment. [Figure 13] This figure shows an example of the functional configuration of the audio-video output system of this embodiment. [Figure 14] This figure shows an example of the operation flow of the audio-video output system of this embodiment. [Figure 15] This figure shows an example of the situation during performance according to this embodiment. [Figure 16] This figure shows an example of the animation behavior of the musical instrument object in this embodiment. [Figure 17] This figure shows an example of the display of the instrument object and the image of the operating part in this embodiment. [Figure 18] This figure shows a modified example of the performance situation in this embodiment. [Modes for carrying out the invention]

[0010] The performance system 1 of this embodiment will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below. In all the drawings for explaining the embodiments, those having the same function are denoted by the same reference numerals, and redundant explanations are omitted. Also, "based on XX" as used in the present application means "based at least on XX", and includes cases where it is based on another element in addition to XX. Also, "based on XX" is not limited to the case where XX is directly used, and includes cases where it is based on something obtained by performing operations or processing on XX. "XX" is an arbitrary element (for example, arbitrary information).

[0011] [Embodiment] Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the device configuration of the performance system 1 of the present embodiment. The performance system 1 includes an instrument information generation device 10, an audio-visual output device 20, a storage unit 30, a terminal device 40, and a display system 50.

[0012] The instrument information generation device 10 and the audio-visual output device 20 are connected to the storage unit 30 via a network N1, and can exchange information with each other. The instrument information generation device 10 and the audio-visual output device 20 are connected to the terminal device 40 via a network N2, and can exchange information with each other. The instrument information generation device 10 and the audio-visual output device 20 are connected to the display system 50 via a network N3, and can exchange information with each other. All of the networks N1 to N3 can exchange information with each other by wireless communication or wired communication. Hereinafter, the description of the exchange of information between the devices via the networks N1 to N3 will be omitted.

[0013] The instrument information generation device 10 and the audio-visual output device 20 are computer devices such as personal computers. In this embodiment, the instrument information generation device 10 and the sound and video output device 20 are described separately, assuming they are individual computer devices. However, this is not limited to the instrument information generation device 10 and the sound and video output device 20, which may be implemented by a single computer device.

[0014] The storage unit 30 is composed of, for example, a hard disk drive or semiconductor memory (flash memory, RAM, ROM), and stores various types of information, such as programs and data read by the instrument information generation device 10 and the sound and video output device 20. The storage unit 30 may also be implemented by a virtual storage device such as a cloud server located outside the instrument information generation device 10 and the sound and video output device 20.

[0015] The terminal device 40 is, for example, a computer device such as a smartphone, tablet, or personal computer, and includes an operation detection unit 41 and a terminal display unit 42. The operation detection unit 41 is equipped with an operation device such as a touch panel, mouse, or keyboard, and detects user U's operations. The terminal display unit 42 includes, for example, a display device such as a liquid crystal display, and presents various images to the user U. In one example of this embodiment, the terminal device 40 is configured as a so-called drawing tablet device and has the function of transmitting a drawing figure P1 drawn by user U to the musical instrument information generation device 10.

[0016] The display system 50 includes a display device 51, an imaging unit 52, and a sound output unit 53. The display device 51, for example, is equipped with a liquid crystal display and displays various images based on image information output by the instrument information generation device 10 and the sound and video output device 20. In one example of this embodiment, the display device 51 consists of two displays, a first display device 511 and a second display device 512, arranged side by side next to each other. The number of display devices 51 may be one or three or more.

[0017] The imaging unit 52, for example, is equipped with a video camera and captures images of the area within the detection area 60 in front of the display device 51. The imaging unit 52 transmits the captured video images to the sound and video output device 20.

[0018] The sound output unit 53 is equipped with, for example, an audio device such as an amplifier or a speaker, and outputs sound based on sound information transmitted from the sound-video output device 20.

[0019] In the following description, the combination of the instrument information generation device 10, the storage unit 30, and the terminal device 40 will also be referred to as the instrument information generation system 11. Furthermore, the combination of the sound and video output device 20, the storage unit 30, and the display system 50 will also be referred to as the sound and video output system 12. Furthermore, the instrument information generation system 11 may include a display system 50, and the sound and video output system 12 may include a terminal device 40.

[0020] The instrument information generation system 11 and the sound and video output system 12 will be described below, in that order. First, the functional configuration and operation flow of the instrument information generation system 11 will be explained with reference to Figures 2 and 3.

[0021] [About the instrument information generation system] Figure 2 shows an example of the functional configuration of the instrument information generation system 11 of this embodiment. Figure 3 shows an example of the operation flow of the instrument information generation system 11 of this embodiment.

[0022] The instrument information generation device 10 includes a calculation unit 100. The arithmetic unit 100 includes, for example, a central processing unit (CPU), and operates based on programs and data stored in the storage unit 30, providing various functions.

[0023] The calculation unit 100 includes, as its functional units, a drawing operation reception unit 110, a type determination unit 120, a tone adjustment unit 130, a frequency assignment unit 140, a musical instrument information generation unit 150, and an object addition unit 160.

[0024] (Step S110) User U draws on the operation detection unit 41 (for example, the touch panel) of the terminal device 40. Drawing means that User U draws any shape. The operation of drawing a shape on the operation detection unit 41 is also called a drawing operation.

[0025] Figure 4 shows an example of the correspondence between the drawing trajectory OT1 and the drawn figure P1 in this embodiment. User U draws the drawing trajectory OT1 shown in the figure on the operation detection unit 41 (for example, a touch panel) using a stylus or finger. The operation detection unit 41 obtains the coordinates of the drawing trajectory OT1 on the touch panel. The terminal display unit 42 displays the drawing figure P1 corresponding to the drawing trajectory OT1 detected by the operation detection unit 41.

[0026] In one example of this embodiment, the performance system 1 targets kindergarten or nursery school children, or elementary school students. Users U in this target group can quickly draw familiar shapes if they are shapes that can be drawn in a single stroke. The instrument information generation device 10 of this embodiment acquires a single-stroke drawing trajectory as a drawing trajectory OT1. In this example, user U draws a heart-shaped drawing trajectory OT1. In this case, the terminal display unit 42 displays the first drawing figure P11 (for example, a heart-shaped figure) shown in the figure. The heart-shaped figure is an example of a single-stroke drawing figure.

[0027] The terminal device 40 transmits the coordinate information of the drawing trajectory OT1 detected by the operation detection unit 41 to the instrument information generation device 10. The drawing operation reception unit 110 of the instrument information generation device 10 acquires coordinate information of the drawing trajectory OT1 transmitted from the terminal device 40. The acquisition of coordinate information of the drawing trajectory OT1 can also be described as the drawing operation reception unit 110 accepting a drawing operation from user U.

[0028] In other words, the drawing operation reception unit 110 receives drawing operations from user U.

[0029] As shown in the figure, the coordinates of the drawing trajectory OT1 and the drawn shape P1 are approximately the same. In the following explanation, when the drawing trajectory OT1 and the drawn shape P1 are not distinguished, they will be collectively referred to as the drawn shape P1 drawn by user U (or simply as the drawn shape P1).

[0030] [Determine the type of instrument] (Step S120) Returning to Figure 3, the type determination unit 120 determines the type of instrument based on the drawing shape P1 drawn by the user U. The correspondence between the drawing shape P1 and the type of instrument will be explained below.

[0031] Figure 5 shows an example of the judgment space for the drawn figure P1 in this embodiment. The judgment space for the drawn figure P1 is composed of coordinate axes that classify the shape of the drawn figure P1 (in this example, a figure drawn in a single stroke) drawn by user U into multiple types.

[0032] In one example of this embodiment, the judgment space of the drawn figure P1 is composed of two coordinate axes: the curvature axis AX1 and the complexity axis AX2.

[0033] The curvature axis AX1 is an axis that indicates whether the curvature of the lines constituting the drawn figure P1 is large (more curved components) or small (fewer curved components and closer to a straight line). The curvature of the lines constituting the drawn figure P1 can be calculated by various algorithms. For example, the type determination unit 120 divides the trajectory of the lines constituting the drawn figure P1 (i.e., the drawing trajectory OT1) into infinitesimal intervals, and calculates the curvature of the lines constituting the drawn figure P1 by accumulating (or integrating) the curvature of each infinitesimal interval along the trajectory.

[0034] In the following explanation, the curvature of the lines that make up the drawn figure P1 will also simply be referred to as the curvature of the drawn figure P1.

[0035] The complexity axis AX2 indicates whether the complexity of the lines constituting the drawn figure P1 is large (more complex shape) or small (simpler shape). The complexity of the lines constituting the drawn figure P1 can be calculated using various algorithms. For example, the type determination unit 120 calculates the complexity of the lines constituting the drawn figure P1 based on the ratio of the distance from the start point to the end point of the lines constituting the drawn figure P1 to the length of the lines constituting the drawn figure P1.

[0036] Generally, the ratio of the distance from the start point to the end point of a line to the length of the line can also be said to be the ratio of the shortest distance between the start and end points to the length of the trajectory when moving along the line (i.e., the distance traveled). If the shortest distance and the distance traveled for a given line are approximately equal, then the line can be said to represent a trajectory that moves along the shortest route, and is considered to have the simplest shape. Conversely, if the shortest distance and the distance traveled for a given line are different, then the line can be said to represent a trajectory that moves along a route that is not the shortest route, and is considered to have a complex shape. Furthermore, if the ratio of the shortest distance to the distance traveled (where the shortest distance is the denominator and the distance traveled is the numerator) is larger for a given line, then the line can be said to have a more complex shape. In other words, the ratio (or proportion) of the shortest distance to the total distance traveled along a given line can be said to indicate the complexity of that line.

[0037] In other words, the type determination unit 120 determines the type of musical instrument based on the curvature of the lines that make up the drawn figure P1 (figure) and the complexity of the drawn figure P1 (figure), which is calculated based on the ratio of the distance from the start point to the end point of the lines that make up the drawn figure P1 (figure) to the length of the lines.

[0038] In the following explanation, the complexity of the lines that make up the drawn figure P1 will also simply be referred to as the complexity of the drawn figure P1.

[0039] [Complexity calculation algorithm (variant)] The complexity calculation algorithm described above is just one example and is not limited to it. For example, complexity may be calculated as follows:

[0040] (Variation 1) (1) Calculate the rectangular frame (outer frame) that circumscribes the drawing shape P1. (2) The length of the circle (or arc) inscribed in the outer frame is used as the reference length for calculation. For example, if the outer frame is a square, the length (circumference) of the circle inscribed in the outer frame is calculated and used as the reference length. For example, if the outer frame is a rectangle, the length of the arc inscribed in the outer frame is calculated and used as the reference length. (3) Compare the reference length calculated in (2) above with the length of the drawing trajectory OT1 of the drawn figure P1. A smaller ratio of the length of the drawing trajectory OT1 to the reference length indicates lower complexity, while a larger ratio indicates higher complexity. The ratio of the reference length to the length of the drawing trajectory OT1 of the drawn figure P1 is calculated as the complexity.

[0041] (Modification 2) (1) Calculate the perimeter (sum of the lengths of each side) of the rectangular frame (outer frame) that circumscribes the drawn shape P1. (2) The perimeter calculated in (1) above is used as the reference length, and the reference length is compared with the length of the drawing trajectory OT1 of the drawn figure P1. A small ratio of the length of the drawing trajectory OT1 to the reference length indicates low complexity, while a large ratio indicates high complexity. The ratio of the reference length to the length of the drawing trajectory OT1 of the drawn figure P1 is calculated as the complexity.

[0042] In the above-described modification 2, the reference length may be the perimeter multiplied by a predetermined factor based on the curvature of the drawing trajectory OT1 of the drawn figure P1. For example, if the curvature of the drawing trajectory OT1 is small (there are many straight parts), the reference length may be the perimeter multiplied by 1 / 2. If the curvature of the drawing trajectory OT1 is large (there are many curved parts), the perimeter may be used as the reference length.

[0043] Alternatively, the maximum (or minimum) complexity among those calculated in parallel by each of the above calculation algorithms may be adopted as the complexity of the drawn figure P1.

[0044] [Selection of instrument type] In one example of this embodiment, the type determination unit 120 selects the type of instrument from among several types of instruments that are pre-stored in association with curvature and complexity.

[0045] More specifically, the type determination unit 120 calculates the curvature and complexity of the drawn figure P1. The type determination unit 120 determines which coordinates in the determination space, indicated by the curvature axis AX1 and the complexity axis AX2, the calculated curvature and complexity of the drawn figure P1 are located at. The type determination unit 120 selects the type of musical instrument that corresponds to the coordinates where the curvature and complexity of the drawn figure P1 are located. An example of the correspondence between the shape of the drawn figure P1 and the type of musical instrument will be explained with reference to Figure 6.

[0046] Figure 6 shows an example of the correspondence between the shape of the drawn figure P1 and the type of musical instrument in this embodiment. The type determination unit 120 selects an instrument from among the multiple instruments shown in the figure that corresponds to the coordinates of the shape of the drawn figure P1 (for example, curvature and complexity).

[0047] In general, the so-called "Bouba / Kiki" effect is cited in psychology as an example of "sound symbolism," where sound evokes a specific image. This also means that there is a correspondence between the visual impression of a shape and the characteristics of the sound (for example, frequency, harmonic composition, degree of reverberation, volume, etc.). In other words, the visual impression of a shape changes with parameters such as the curvature and complexity of the shape.

[0048] For example, different types of instruments produce various characteristics of sound, such as different frequencies, harmonic components, and envelope curves that show the time-dependent changes in sound intensity. These differences in sound characteristics result in different shapes of figures evoked by the sound symbolism in the user U. As an example, as shown in the figure, in the case of a simple drawing shape P1 with small curvature and linearity, one can imagine a xylophone that produces sharp, linear, and relatively simple sounds. The type determination unit 120 selects an instrument that produces the sound symbolized by the drawing figure P1 drawn by user U, based on the sound symbolism theory described above.

[0049] In other words, the type determination unit 120 determines the type of musical instrument based on the shape of the drawing figure P1 (figure) drawn by the drawing operation.

[0050] [Mapping musical scales to instrument objects] (Step S130) Returning to Figure 3, the instrument information generation unit 150 generates the shape of the instrument object P2 based on the drawing shape P1 drawn by the user U.

[0051] Figure 7 shows an example of the correspondence between the drawing figure P1 and the instrument object P2 in this embodiment. The figure also shows an example of the correspondence between the first drawing figures P11 to the fifth drawing figures P15 and the first instrument objects P21 to the fifth instrument objects P25.

[0052] The instrument information generation unit 150 generates the shape of the instrument object P2 by making the shape of the instrument object P2 similar to the shape of the drawing trajectory OT1 of the drawing figure P1 (for example, a line drawn in a single stroke). With the instrument information generation device 10 configured in this way, user U can recognize that the instrument object P2 is not something given by a computer device, but something that they created themselves. As a result, user U can enhance their sense of ownership and familiarity with the instrument object P2.

[0053] Furthermore, User U's sense of ownership and affinity towards the instrument object P2 refers to User U's desire to possess the instrument object P2, similar to the feelings of "cute" or "fun to be with" that one might have towards a pet they own. In the following explanation, these feelings of User U towards the instrument object P2 will be collectively referred to as "liking for the instrument object P2" (or simply "liking").

[0054] (Step S140) Returning to Figure 3, the object attachment unit 160 attaches predetermined objects to the instrument object P2. These predetermined objects include an eye object OBJ1 and a mouth object OBJ2. In the following description, the predetermined object that the object attachment unit 160 attaches to the instrument object P2 is also referred to as the attached object.

[0055] Figure 8 shows an example of an object attached to the instrument object P2 in this embodiment. By attaching the eye object OBJ1 and the mouth object OBJ2 to the instrument object P2, the instrument object P2 is personified. As a result, the user U's sense of ownership and familiarity with the instrument object P2 can be enhanced.

[0056] The added objects (for example, the eye object OBJ1 and the mouth object OBJ2) are designed with shapes and positions that enhance the appeal of the instrument object P2. An example of an algorithm for placing added objects is described below.

[0057] The object attachment part 160 sets a reference position for the drawing shape P1. Since the drawing shape P1 and the instrument object P2 are similar in shape, it can also be said that "the object attachment part 160 sets a reference position for the instrument object P2." Similarly, in the following explanation, any mention of the drawing shape P1 can be replaced with the instrument object P2.

[0058] The reference position should preferably be a position that can be geometrically derived in common from various shapes of drawn figures P1. For example, the reference position is the centroid of the area enclosed by the drawing trajectory OT1 of the drawn figure P1. In this case, the object attachment unit 160 sets the centroid of the area of ​​the drawn figure P1 (the centroid CG1 as exemplified in the figure) as the reference position.

[0059] The object attachment unit 160 places the attached object on the circumference of a virtual reference circle (reference circle CL1 as exemplified in the figure) with radius r centered on a reference position (for example, the centroid CG1).

[0060] In other words, the object attachment unit 160 attaches the eye object OBJ1 and the mouth object OBJ2 at positions based on the reference position of the drawing shape P1 (shape).

[0061] With the instrument information generation device 10 configured in this way, even if the shapes of the drawn figures P1 are all different, additional objects can be placed based on a predetermined placement algorithm.

[0062] Note that for the eye object OBJ1 and mouth object OBJ2, objects with shapes corresponding to the shape of drawing shape P1 may be selected.

[0063] Figure 9 shows an example of the correspondence between the shape of the drawing figure P1 in this embodiment and the object attached to the instrument object P2. The figure shows an example where the shape of the attached object is associated with the position of the shape of the drawing figure P1 on the curvature axis AX1.

[0064] The closer the position of the drawing shape P1 on the curvature axis AX1 is to a straight line, the harder (sharper) the outlines of the eye object OBJ1 and mouth object OBJ2 will be. The closer it is to a curve, the softer (rounder) the outlines of the eye object OBJ1 and mouth object OBJ2 will be.

[0065] Additional objects may also be displayed with animation. Figures [A] and [B] show examples of how additional objects can be displayed with animation. For example, the eye object OBJ1 can be animated to blink, move the pupil, etc. The mouth object OBJ2 can be animated to open and close its mouth, etc.

[0066] Furthermore, the shape of the added object may be associated with the complexity axis AX2. Alternatively, the shape of the added object may be associated with a combination of the curvature axis AX1 and the complexity axis AX2. In other words, the shape of the added object is determined based on at least one of the curvature axis AX1 and the complexity axis AX2.

[0067] In other words, the object attachment unit 160 attaches the eye object OBJ1 representing the character's eyes and the mouth object OBJ2 representing the character's mouth to the drawing shape P1 (shape) at positions based on the reference position of the drawing shape P1 (shape) by selecting the pattern of the eye object OBJ1 representing the character's eyes and the pattern of the mouth object OBJ2 representing the character's mouth, based on at least one of the curvature of the lines constituting the drawing shape P1 (shape) drawn by the drawing operation and the complexity of the drawing shape P1 (shape) calculated based on the ratio of the distance from the start point to the end point of the lines constituting the drawing shape P1 (shape) to the length of the lines.

[0068] Note that the mapping of the shape of an added object to the curvature axis AX1 or the complexity axis AX2 is just one example, and is not limited to this. The shape of an added object may be determined based on some other determination axis other than the curvature axis AX1 or the complexity axis AX2 corresponding to the shape of the drawn figure P1.

[0069] Returning to Figure 3, the frequency assignment unit 140 assigns frequencies corresponding to musical scales to the shape of the instrument object P2 generated by the instrument information generation unit 150. A specific example of the frequency assignment procedure by the frequency assignment unit 140 will be described below.

[0070] (Step S150) The frequency allocation unit 140 divides the drawn figure P1 into multiple regions R.

[0071] Figure 10 shows an example of the correspondence between the region R of the drawing figure P1 in this embodiment and a musical scale. In this example, the drawing figure P1 is shown as the first drawing figure P11 (for example, a heart-shaped figure).

[0072] The frequency allocation unit 140 unfolds the outline of the drawn figure P1 (figure) in a predetermined direction and divides the unfolded drawn figure P1a.

[0073] In the example shown in the figure, the frequency allocation unit 140 unfolds the drawn figure P1 in the Xp1 axis direction and divides the unfolded drawn figure P1a into eight parts in the Xp1 axis direction. When the drawn figure P1a is divided in the Xp1 axis direction, eight regions R1 to R8 are formed.

[0074] (Step S160) The frequency assignment unit 140 assigns frequencies according to the musical scale for each region R.

[0075] In other words, the frequency assignment unit 140 assigns a frequency corresponding to the musical scale to each region R into which the drawn shape P1 (shape) is divided.

[0076] The frequency of the note A4, according to the so-called international musical notation, is 440.000 Hz. Similarly, the frequency of the note C4 is 261.626 Hz. The frequency of the note C5 is 523.251 Hz. Note that these frequencies are just examples when the reference tone (A4) is set at 440.000 Hz, and for a typical musical scale (a frequency obtained by dividing an octave into 12 notes), the frequencies may shift to higher or lower frequencies.

[0077] In this example, the frequency assignment unit 140 assigns the frequency of sound C4 (261.626 Hz) to region R1. Similarly, the frequency assignment unit 140 assigns the frequencies of sounds D4 to C5 to regions R2 to R8, respectively.

[0078] In other words, the frequency assignment unit 140 unfolds the outline of the drawn figure P1 (figure) in a predetermined direction and assigns musical scale frequencies to the regions obtained by dividing the unfolded drawn figure P1a.

[0079] Here, the frequency assignment unit 140 assigns the so-called white keys of a piano to region R, but does not assign the black keys. In other words, the frequency assignment unit 140 assigns only the white keys of a piano to region R. In the following explanation, the scale composed of the white and black keys of a piano is also called a diatonic scale (12-tone scale). Of the notes that make up the diatonic scale, the notes corresponding to the white keys of the piano are also called the diatonic group KW, and the notes corresponding to the black keys of the piano are also called the derived group KB.

[0080] In other words, the frequency assignment unit 140 assigns the frequencies of eight consecutive notes included in the diatonic group KW, which constitutes the diatonic scale, to the region R.

[0081] Generally, when sounds from the derived sound group KB are pronounced simultaneously with other sounds, the resulting sound can feel unnatural (for example, like a dissonance), compared to when multiple sounds from the diatonic sound group KW are pronounced simultaneously. In this embodiment, the frequency assignment unit 140 excludes the derived sound group KB from the frequencies assigned to the region R, thereby reducing the unnatural feeling when multiple frequencies are pronounced simultaneously.

[0082] [Examples of frequency allocation]

[0083] (1) Modification of the range of musical scale assignment to region R In the example above, the note C4 is assigned to region R1, where the lowest frequency is assigned, and the note C5 is assigned to region R8, where the highest frequency is assigned. In other words, the eight consecutive notes from region R1 to region R8 constitute the so-called C major scale, but this is not the only example. Depending on the type of instrument, differences in the range of reproducible notes can sometimes result in an unnatural sound when using the C major scale. Therefore, the frequency assignment unit 140 assigns the frequencies of the scale within the range where the sound resonance is natural according to the type of instrument to the region R. For example, the frequency assignment unit 140 may assign the eight notes from A3 to A4 to regions R1 to R8. In this case, the eight consecutive notes from region R1 to region R8 constitute the scale of A minor in the so-called natural minor scale.

[0084] (2) Modified shape of region R Figure 11 shows an example of the correspondence between the region R of the drawing figure P1 in this embodiment and a musical scale. In this example, the drawing figure P1 is shown as the first drawing figure P11 (for example, a heart-shaped figure). In this modified example, the frequency allocation unit 140 divides the drawn figure P1 in a predetermined direction without unfolding the drawn figure P1 (i.e., while maintaining the shape of the drawn figure P1).

[0085] The frequency allocation unit 140 divides the drawn figure P1 in a predetermined direction. In the example shown in the figure, the frequency allocation unit 140 divides the drawn figure P1 into eight sections along the Xp1 axis. When the drawn figure P1 is divided along the Xp1 axis, eight regions R1 to R8 are formed.

[0086] In other words, the frequency assignment unit 140 assigns the frequencies of a musical scale to a region R obtained by dividing the drawn figure P1 (figure) into eight parts in a predetermined direction. This scale is created by dividing the area between a reference tone corresponding to the type of instrument and a tone one octave away from the reference tone into eight parts. In this example, the region R is divided into eight regions R1 to R8, but the number of divisions is not limited to eight. The frequency assignment unit 140 assigns the frequencies of the divided scale to the regions R, which are divided into a number of divisions corresponding to the number of notes in a predetermined scale, between a reference tone corresponding to the type of instrument and a tone that is a predetermined frequency away from the reference tone according to the number of divisions mentioned above. In other words, eight divisions are just one example of the number of divisions mentioned above.

[0087] With the instrument information generation device 10 configured in this way, the instrument object P2 can be likened to instruments that express musical scales, such as keyboard instruments like pianos and clavinets, keyboard percussion instruments like xylophones and glockenspiels, wind instruments like flutes and oboes, and string instruments like violins and guitars. Therefore, the instrument information generation device 10 makes it possible to express sound effects produced by various playing techniques characteristic of these instruments (for example, glissando).

[0088] (3) Modified examples of the direction of division of region R In the example described above, the direction of division of region R was assumed to be along the xp1 axis, but this is not the only option. The direction of division of region R may also be along the yp1 axis. Furthermore, region R may be divided into a matrix combining the xp1 axis and the yp1 axis.

[0089] In either frequency assignment method, the frequency assignment unit 140 assigns only the fundamental tone group KW to region R, and does not assign the derived tone group KB to region R. Since the frequency assignment unit 140 in this embodiment excludes the derived tone group KB from the frequencies assigned to region R, it is possible to reduce the unnatural feeling when multiple frequencies are pronounced simultaneously.

[0090] (Step S170) Returning to Figure 3, the tone adjustment unit 130 adjusts the tone according to the type of instrument selected in step S120 based on various conditions. As an example, the tone adjustment unit 130 adjusts the tone according to the drawing speed of the drawing figure P1 by the user U. The drawing speed of the drawing figure P1 refers to the drawing time required to draw the drawing trajectory OT1 from the start point to the end point, or the ratio of the length of the line of the drawing trajectory OT1 from the start point to the end point to the drawing time.

[0091] In other words, the tone adjustment unit 130 adjusts the tone of the instrument based on the drawing speed of the drawing shape P1 (shape) created by the drawing operation.

[0092] For example, a short drawing time (i.e., when user U draws quickly) evokes a bright, lively, or hard tone. A long drawing time (i.e., when user U draws slowly and deliberately) evokes a rich, relaxed, or soft tone.

[0093] The tone adjustment unit 130 adjusts the timbre of the instrument to a timbre associated with the drawing speed described above, based on the drawing speed of the drawn figure P1. With the instrument information generation device 10 configured in this way, it is possible to provide the instrument object P2 with a timbre that is in line with the user U's explicit drawing intention, or with sound symbols that appear in the drawn figure P1 without the user U being aware of them.

[0094] [Generating instrument information] (Step S180) The instrument information generation unit 150 generates instrument information D1 by summarizing the parameters of the instrument object P2 generated in steps S120 to S170.

[0095] Figure 12 shows an example of instrument information D1 in this embodiment. Instrument information D1 consists of an instrument information ID that individually identifies the generated instrument object P2, and various parameters. The various parameters include the shape of the instrument object P2, the type of instrument, the position of region R within the instrument object P2, the frequencies assigned to each region R, the timbre adjustment results, and the shape and position of any added objects.

[0096] Shape information D11 is information indicating the shape of the instrument object P2. Instrument type information D12 is information indicating the type of instrument determined by the type determination unit 120. Frequency information D13 is information indicating the frequency assigned to region R by the frequency assignment unit 140. Timbre information D14 is information indicating the timbre adjusted by the timbre adjustment unit 130. The instrument information D1 is composed of the shape information D11, instrument type information D12, frequency information D13, and timbre information D14.

[0097] In other words, the instrument information generation unit 150 generates instrument information D1 by associating shape information D11, which indicates the shape of the drawn figure P1 (figure), instrument type information D12, which indicates the type of instrument that has been determined, and frequency information D13, which indicates the frequency assigned to the region R.

[0098] As described above, instrument information D1 includes timbre information (timbre information D14) adjusted by the timbre adjustment unit 130.

[0099] In other words, the instrument information generation unit 150 further associates the adjusted timbre information to generate instrument information D1.

[0100] Furthermore, as mentioned above, the instrument information D1 may also include additional object information D15 that indicates the shape and position of the additional object.

[0101] In other words, the instrument information generation unit 150 further associates additional object information D15, which represents the eye object OBJ1 and the mouth object OBJ2, to generate instrument information D1.

[0102] (Step S190) Returning to Figure 3, the instrument information generation unit 150 stores the generated instrument information D1 in the storage unit 30, and ends the series of operations for generating the instrument information D1.

[0103] Next, the functional configuration and operation flow of the sound and video output system 12 will be explained with reference to Figures 13 and 14. The sound and video output system 12 is a system that outputs sound and video from an instrument object P2 using the instrument information D1 generated by the instrument information generation device 10 in the flow described above.

[0104] [About the audio / video output system] Figure 13 shows an example of the functional configuration of the audio-video output system 12 of this embodiment. Figure 14 shows an example of the operation flow of the audio-video output system 12 of this embodiment. The sound and video output system 12 comprises a sound and video output device 20, a storage unit 30, a display device 51, an imaging unit 52, and a sound output unit 53. The display device 51, imaging unit 52, and sound output unit 53 are collectively referred to as the display system 50.

[0105] In this embodiment, the sound and video output device 20 and the instrument information generation device 10 are described as separate computer devices, but this is not the only way to describe them. As mentioned above, the sound and video output device 20 and the instrument information generation device 10 may be implemented by a single computer device.

[0106] The memory unit 30 stores the instrument information D1 generated by the instrument information generation device 10. The configuration of the memory unit 30 is the same as that of the memory unit 30 of the instrument information generation system 11, so its explanation is omitted.

[0107] The audio-video output device 20 is a computer device such as a personal computer. The audio-video output device 20 includes an arithmetic unit 200. The arithmetic unit 200 includes, for example, a central processing unit (CPU), and operates based on programs and data stored in the memory unit 30, providing various functions.

[0108] The calculation unit 200 includes, as its functional units, a detection unit 210, an instrument information acquisition unit 220, a display control unit 230, a coordinate calculation unit 240, a contact area determination unit 250, and a sound output unit 260.

[0109] (Step S210) The instrument information acquisition unit 220 acquires instrument information D1 from the storage unit 30. As described above, instrument information D1 is information generated based on the shape of the drawing figure P1 drawn by the user U's drawing operation. Instrument information D1 is information to which shape information D11, instrument type information D12, and frequency information D13 are associated with an instrument ID that identifies the instrument object P2.

[0110] In other words, the instrument information acquisition unit 220 acquires instrument information D1, which is information generated based on the shape of the drawing figure P1 (figure) drawn by the user U's drawing operation, and which is associated with shape information D11 indicating the shape of the drawing figure P1 (figure), instrument type information D12 indicating the type of instrument based on the shape of the drawing figure P1 (figure), and frequency information D13 indicating the frequency assigned to each region R of the drawing figure P1 (figure).

[0111] (Step S220) The display control unit 230 displays an image of the instrument object P2 based on the acquired instrument information D1 on the display device 51.

[0112] Figure 15 shows an example of the situation during performance in this embodiment. The display device 51 displays an image of the instrument object P2.

[0113] In the following description, the coordinate system on the display surface of the display device 51 is also referred to as the display coordinate system (display X coordinate xd, display Y coordinate yd). That is, the display control unit 230 displays the instrument object P2 at a predetermined position in the display coordinate system based on a predetermined display position calculation algorithm.

[0114] User U performs a gesture to operate the instrument object P2 within a detection area 60 set at a predetermined position in front of the display device 51. The gesture to operate the instrument object P2 is also called a performance operation.

[0115] The performance operation involves, for example, the user U moving their right or left hand towards the display device 51 within the detection area 60. In this case, the user U is making movements as if they were a conductor facing the display device 51. The detection area 60 can also be described as the area to be captured by the imaging unit 52. In other words, the user U performs the performance operation within the area to be captured by the imaging unit 52. Next, a specific example of the display of the instrument object P2 by the display control unit 230 will be described.

[0116] [Animation behavior of instrument object P2] Figure 16 shows an example of the animation operation of the instrument object P2 in this embodiment. The display control unit 230 can animate the instrument object P2 based on a predetermined display algorithm.

[0117] (1) Animation display of added objects The display control unit 230 animates the added objects. These added objects include, for example, an eye object OBJ1 and a mouth object OBJ2. The display control unit 230 makes the eye object OBJ1 blink and move the position of its pupils. The display control unit 230 also makes the mouth object OBJ2 open and close.

[0118] With the sound and video output device 20 configured in this way, the expressions of the instrument object P2 can be enriched, and the user U's favorability towards the instrument object P2 can be increased.

[0119] Furthermore, the display control unit 230 may display animations of the additional objects at a speed and frequency corresponding to the performance operation. With the sound and video output device 20 configured in this way, the instrument object P2 appears to be responding to the user U's performance operation, thereby enhancing the interactivity that user U perceives with the instrument object P2.

[0120] (2) Movement of the display position of instrument object P2 The display control unit 230 may move the display position of the instrument object P2 on the display screen of the display device 51 over time. For example, the display control unit 230 moves the instrument object P2 along a movement trajectory OT2. The movement trajectory OT2 is, for example, defined within the display range of the display device 51, such that the angle of incidence and the angle of emission are the same (i.e., corresponding to the reflection properties of physical objects in real space).

[0121] In other words, the display control unit 230 changes the position of the instrument object P2 on the display screen of the display device 51.

[0122] By having the instrument object P2 move along the movement trajectory OT2, the gameplay of playing the instrument is improved compared to when the instrument object P2 is stationary on the display screen, providing a greater sense of enjoyment for the user U.

[0123] Furthermore, the movement trajectory OT2 may be such that the angle of incidence and the angle of departure are different from each other (i.e., contrary to the properties of reflection of an object). If the reflection properties of an object are reversed, it can create a sense of surprise for the user U regarding the movement trajectory OT2 of the instrument object P2, improving the gameplay aspect of the performance and providing the user U with greater enjoyment.

[0124] [Detecting User U's posture] (Step S230) Returning to Figure 14, the imaging unit 52 captures an image of user U within the detection area 60. The imaging unit 52 transmits the captured video image of user U to the sound and video output device 20.

[0125] The detection unit 210 of the sound and video output device 20 acquires a video image of user U received from the imaging unit 52. The detection unit 210 detects the user U's operating area OP from the acquired video image. The operating area OP is a predetermined part of user U's body.

[0126] The detection of the user U's operating part OP by the detection unit 210 is also referred to as detecting the user U's posture. The detection unit 210 detects the user U's posture without using devices that come into contact with the user U and detect user U's operations, such as a keyboard, mouse, joystick, handheld controller, or foot sensor.

[0127] In other words, the detection unit 210 detects the user U's posture without contact. Non-contact detection includes methods that use video of the user U captured by the imaging unit 52, as well as methods that use motion capture sensors such as infrared sensors and ultrasonic sensors.

[0128] In the following description, the coordinate system used by the detection unit 210 to detect the position of the operating part OP is also referred to as the real-space coordinate system (real-space X coordinate xr, real-space Y coordinate yr). The real-space coordinate system is a coordinate system that indicates the position within the plane set in the detection area 60. The real-space X coordinate xr is set parallel to the display X coordinate xd, and the real-space Y coordinate yr is set parallel to the display Y coordinate yd. In other words, the plane formed by the display coordinate system and the plane formed by the real-space coordinate system are set parallel to each other. It should be noted that "parallel" in this context does not necessarily mean mathematically parallel; it is sufficient that the position of the user U's operating part OP in the real-space coordinate system can be converted to the display position on the display device 51 in the display coordinate system.

[0129] (1) When there is only one operating site OP The operating area OP may be any single specific part of the user U's body. For example, the operating area OP may be only the fingertips of the user U's right hand (first operating area OP1).

[0130] (2) When there are multiple operating sites OP The operating area OP does not necessarily have to be a single specific location. The operating area OP may be multiple parts of the user U's body. For example, as shown in Figure 15, the operating area OP may be the fingertips of the user U's right hand (first operating area OP1) or the fingertips of the left hand (second operating area OP2).

[0131] In other words, the operating area OP includes a first operating area OP1 and a second operating area OP2, each corresponding to multiple parts of the user U's body.

[0132] In the example described above, the detection unit 210 is described as detecting a predetermined part (for example, a fingertip) as the operating part OP, but it is not limited to this. For example, the detection unit 210 may be capable of detecting the speed of movement of an object in the detection area 60. In this case, the detection unit 210 may detect the part of the user U's body with the fastest movement speed as the operating part OP. Furthermore, the detection unit 210 may be capable of detecting the position in the depth direction within the detection area 60. In this case, the detection unit 210 may detect the part of the user U's body closest to the display device 51 (or imaging unit 52) ​​as the operating part OP.

[0133] [Display of images of the operated area] Figure 17 shows an example of the display of the instrument object P2 and the operation part image P3 in this embodiment. The operation part image P3 is an image showing the position of the operation part OP on the display screen of the display device 51. The figure shows the positional relationship between two types of operation positions PT (operation position PT1, operation position PT2) and the instrument object P2 on the display screen of the display device 51. The first operation part image P31 shown in the figure is the operation part image P3 showing operation position PT1. The second operation part image P32 is the operation part image P3 showing operation position PT2.

[0134] (Step S240) Returning to Figure 14, the coordinate calculation unit 240 calculates the coordinates (display X coordinate xd, display Y coordinate yd) of the operation part image P3 to be displayed on the display screen of the display device 51, based on the posture of the user U detected by the detection unit 210. The coordinates of the operation part image P3 on the display screen of the display device 51 are also called operation coordinate information D2. In other words, the coordinate calculation unit 240 calculates the operation coordinate information D2 by converting the coordinates of the operation part OP in the real space coordinate system to the coordinates in the display coordinate system. That is, the coordinates of the operation part image P3 (operation coordinate information D2) are obtained by converting the coordinates of the operation part OP in the real space coordinate system to the coordinates in the display coordinate system.

[0135] In other words, the coordinate calculation unit 240 calculates the coordinates of the operation part image P3 displayed on the display screen of the display device 51 as operation coordinate information D2, based on the detected posture.

[0136] The display control unit 230 displays the operation part image P3 corresponding to the operation part OP detected by the detection unit 210 on the display device 51. At this time, the display control unit 230 displays the operation part image P3 and the instrument object P2 simultaneously on the display device 51. In other words, the display control unit 230 displays the instrument object P2 and the operation part image P3 superimposed on the display screen of the display device 51.

[0137] In other words, the display control unit 230 superimposes an instrument object P2, which is an image based on the acquired instrument information D1, and an operation part image P3, which is an image based on the detected posture and shows an operation part OP that is at least a part of the user U's body, and displays them on the display screen of the display device 51.

[0138] In the following explanation, the position of the operating part OP on the display screen of the display device 51 is also referred to as the operating position PT. That is, the operating part image P3 is an image that shows the operating position PT. When the user U changes their posture (for example, when they move the operating part OP), the display control unit 230 changes the display position of the operating part image P3 in accordance with the change in posture.

[0139] Up to this point, the display control unit 230 has been described as displaying the operating part OP, which is a part of the user U's body, on the display screen, but it is not limited to this. The display control unit 230 may also superimpose the image of the user U (for example, an image of the user U's whole body) captured by the imaging unit 52 in step S230 onto the image of the operating part OP and display it. With the sound and video output device 20 configured in this way, the user U is displayed on the display screen together with the instrument object P2, which can further enhance the user U's sense of ownership and familiarity with the instrument object P2.

[0140] [Interactive display of instrument object P2 based on user U's posture] Up to this point, we have described the case in which the display position of the instrument object P2 moves based on a predetermined movement trajectory OT2. In other words, we have described how the display control unit 230 changes the position of the instrument object P2 on the display screen of the display device 51 in a manner that is not based on the posture of the user U.

[0141] The instrument object P2 may move in response to the user U's playing operations. In this case, the display control unit 230 calculates the movement trajectory OT2 of the instrument object P2 each time (i.e., in real time) based on the user U's posture detected by the detection unit 210.

[0142] In other words, the display control unit 230 changes the position of the instrument object P2 on the display screen of the display device 51 in a manner that is based on the posture of the user U.

[0143] For example, the display control unit 230 causes the instrument object P2 to perform a predetermined movement when user U assumes a specific posture (pose). For example, when user U beckons or makes a heart shape with their fingers, the display control unit 230 moves the instrument object P2 closer to the display position of user U's control part OP.

[0144] In other words, when the user U is in a predetermined posture, the display control unit 230 changes the position of the instrument object P2 on the display screen of the display device 51 to bring it closer to the operation position PT.

[0145] For example, user U may be shorter than the display device 51. In such a case, if the instrument object P2 is above the display device 51 (in the direction where the display Y coordinate yd is smaller), user U's operating part OP may not be able to reach the display position of the instrument object P2. With the sound and video output device 20 configured as described above, when user U assumes a specific posture, the instrument object P2 approaches the control part OP, allowing user U to operate the instrument object P2 even if they are relatively short. With the sound and video output device 20 configured in this way, the game-like nature of the performance operation is improved, providing user U with greater enjoyment.

[0146] [Collision detection for instrument object P2] (Step S250) The contact area determination unit 250 determines a collision between the operating part OP and the instrument object P2 based on the position of the operating part OP on the display screen of the display device 51 and the display position of the instrument object P2.

[0147] In the example shown in the figure, the first operating area image P31 displayed at operating position PT1 is not in contact with the instrument object P2. The second operating area image P32 displayed at operating position PT2 is in contact with the instrument object P2. In this example, the contact area determination unit 250 determines that operating position PT1 is not in contact with the instrument object P2, and that operating position PT2 is in contact with the instrument object P2.

[0148] More specifically, the contact area determination unit 250 calculates the coordinates of the outer shape of the instrument object P2 in the display coordinate system based on the shape of the outer shape of the instrument object P2 indicated by the shape information D11 included in the instrument information D1 and the position of the instrument object P2 in the display coordinate system in which it is displayed. The contact area determination unit 250 compares the coordinates of the calculated outline of the instrument object P2 with the coordinates of the operating area image P3. The contact area determination unit 250 determines that the instrument object P2 and the operating area image P3 are in contact if the coordinates of the operating area image P3 coincide with any position on the outline of the instrument object P2.

[0149] As described above, the instrument object P2 is divided into multiple regions R. The instrument information D1 includes shape information D11 that shows the shape of the regions R. The contact region determination unit 250 refers to the shape information D11 and determines which of the multiple regions R of the instrument object P2 the operating part image P3 has come into contact with. Among the multiple regions R of the operating coordinate information D2, the region R that the operating part image P3 (i.e., the operating position PT of user U) has come into contact with is also called the contact region RT.

[0150] In the example shown in the figure, the second operating area image P32 is in contact with region R2 of the instrument object P2. In this case, the contact region determination unit 250 determines that the operating area image P3 is in contact with region R2.

[0151] In other words, the contact area determination unit 250 determines, based on the operation coordinate information D2 and the shape information D11 of the instrument object P2, the contact area RT among the multiple areas R of the instrument object P2 displayed on the display screen of the display device 51, which is the area R that the operation position PT on the display screen of the display device 51, indicated by the operation coordinate information D2, has come into contact with.

[0152] As mentioned above, the operating area OP does not necessarily have to be a single specific location. For example, if the fingertips of the user U's right hand (first operating area OP1) and the fingertips of the left hand (second operating area OP2) are both operating areas OP, the contact area determination unit 250 determines the contact area RT corresponding to the first operating area OP1 and the contact area RT corresponding to the second operating area OP2, respectively.

[0153] [Sound output] (Step S260) Returning to Figure 14, the sound output unit 260 outputs a sound to the sound output unit 53 at the frequency assigned to the contact area RT determined in step S250. As a result, the sound output unit 53 outputs a sound at the frequency assigned to the contact area RT to the user U.

[0154] Furthermore, if there are multiple operating parts OP, such as the fingertips of the user U's right hand (first operating part OP1) and the fingertips of the left hand (second operating part OP2), the sound output unit 260 outputs a sound with a frequency assigned to the contact area RT corresponding to the first operating part OP1, and a sound with a frequency assigned to the contact area RT corresponding to the second operating part OP2, respectively.

[0155] As described above, each region R of the instrument object P2 is assigned a frequency corresponding to a musical scale. Therefore, depending on which of the multiple regions R (e.g., regions R1 to R8) the manipulated area image P3 touches, a sound with a frequency corresponding to the region R is output. In other words, the instrument object P2 functions as an instrument that plays a musical scale according to the region R (contact region RT) that the manipulated area image P3 touches.

[0156] Furthermore, the sound output unit 260 outputs a sound with a tone corresponding to the instrument type information D12 included in the instrument information D1 acquired in step S210 to the sound output unit 53.

[0157] In other words, the sound output unit 260 outputs a sound that has a frequency assigned to the determined contact area RT and a timbre corresponding to the type of instrument indicated by the instrument information D1.

[0158] As described above, the type of instrument is selected based on the shape of the drawing P1 created by user U. In other words, the sound and video output device 20 can provide user U with the enjoyment and satisfaction of being able to play an instrument that user U has created themselves on a large screen such as the display device 51.

[0159] [Render sound control] Furthermore, the sound output unit 260 may change the characteristics of the sound based on the user U's posture. For example, the sound output unit 260 may change the frequency (i.e., pitch) based on the user U's right-hand playing operation, and change the reverberation of the sound and the envelope curve showing the time change of volume based on the user U's left-hand playing operation. In this example, the contact area RT is determined using the operating coordinate information D2 corresponding to the first operating part OP1 (e.g., the user U's right hand), and the output frequency is changed accordingly. In other words, in this example, the output frequency is not affected by the second operating part OP2 (e.g., the user U's left hand).

[0160] In other words, the sound output unit 260 changes the output frequency by changing the coordinate of the first operating part OP1, and changes the characteristics of the output sound by changing the coordinate of the second operating part OP2.

[0161] With the sound and video output device 20 configured in this way, it is possible to enrich the performance expression using the instrument object P2.

[0162] Furthermore, the sound output unit 260 may change the characteristics of the sound it outputs depending on the user U's performance operation. For example, the sound output unit 260 may change the characteristics of the sound it outputs depending on the speed of change of the user U's operating part OP, the taking of a predetermined pose, the difference in color of an object the user U is wearing (for example, holding in their hand), etc.

[0163] In other words, the sound output unit 260 changes the mode of the sound it outputs according to at least one of the following: the rate of change of the coordinates of the operating part OP, the shape of the operating part OP, and the color of the operating part OP.

[0164] Furthermore, changes in the characteristics of sound include at least one change in the duration of the sound, the reverberation time of the sound, the shape of the sound's envelope curve, the composition of the sound's harmonic components, the composition of the chords, the degree of distortion of the sound's waveform, and the degree of modulation of the sound.

[0165] (Step S270) The arithmetic unit 200 determines whether it is necessary to end the performance of the instrument object P2. If the arithmetic unit 200 determines that the performance of the instrument object P2 should continue (Step S270; NO), it returns to step S220 and continues the processes described above. If the arithmetic unit 200 determines that the performance of the instrument object P2 should end (Step S270; YES), it terminates the series of sound and video output processes.

[0166] [Variation of audio-video output device 20: Performance by multiple users U] Figure 18 shows a modified example of the performance situation in this embodiment. Up to this point, the explanation has been given using the case where there is one user U as an example, but it is not limited to this. The sound and video output system 12a of this modified example allows multiple users U to operate the performance simultaneously. The figure shows a scenario where two users U, user U1 and user U2, are simultaneously operating the device.

[0167] When multiple users U perform performance operations simultaneously, there are two possibilities: (1) multiple users U perform performance operations on a single instrument object P2 simultaneously, and (2) each user U can view a separate instrument object P2 (i.e., multiple instrument objects P2 are displayed simultaneously).

[0168] (1) When multiple users U simultaneously perform performance operations on a single instrument object P2 In the sound and video output system 12a, in step S230 described above, the detection unit 210 detects the posture of multiple users U in a non-contact manner. In the example shown in the figure, the first operation part OP1 and the second operation part OP2, which are operation parts OP of user U1, and the third operation part OP3 and the fourth operation part OP4, which are operation parts OP of user U2, are detected, respectively.

[0169] In step S230 described above, the coordinate calculation unit 240 calculates the coordinates of the operation part image P3 displayed on the display screen of the display device 51 on the display screen of the display device 51 as operation coordinate information D2 for each user U, based on the detected posture.

[0170] In step S240 described above, the display control unit 230 superimposes an image based on the acquired instrument information D1, which is an instrument object P2 that is an image of a pattern corresponding to the shape of the drawing figure P1 (figure), and an operation part image P3 for each user U that is an image based on the detected posture and shows the operation part OP which is at least a part of the user U's body, and displays them on the display screen of the display device 51. In the example shown in the figure, the display control unit 230 displays the first operation area image P31 and the second operation area image P32 corresponding to the first operation area OP1 and the second operation area OP2 of user U1, and the third operation area image P33 and the fourth operation area image P34 corresponding to the third operation area OP3 and the fourth operation area OP4 of user U2, respectively.

[0171] In step S250 described above, the contact area determination unit 250 determines, for each user U, the contact area RT among multiple areas R of the instrument object P2 displayed on the display screen of the display device 51, which is the area R that the operation position PT on the display screen of the display device 51, indicated by the operation coordinate information D2, has come into contact with, based on the operation coordinate information D2 and the shape information D11 of the instrument object P2.

[0172] In other words, the sound output unit 260 outputs a sound that, for each contact area RT determined for each user U, has a frequency assigned to the contact area RT and a timbre corresponding to the type of instrument indicated by the instrument information D1. In this example, if there are multiple contact areas RT, the sound output unit 260 simultaneously outputs sounds of multiple frequencies corresponding to the contact areas RT. While only single notes can be output when one user U plays, the sound and video output system 12a configured in this way enables the output of chords when multiple users U play simultaneously. In other words, the sound and video output system 12a configured in this way makes it possible to differentiate the sound output when one user U plays from the sound output when multiple users U play simultaneously. Therefore, the sound and video output system 12a can give users U the enjoyment of playing together in an ensemble.

[0173] In other words, the contact area determination unit 250 determines multiple contact areas RT where multiple operating positions PT, indicated by the operating coordinate information D2 of multiple users U, come into contact with a single instrument object P2 displayed on the display screen of the display device 51. The sound output unit 260 outputs sounds corresponding to multiple contact areas RT determined for each user U for a single instrument object P2.

[0174] (2) When assigning each instrument object P2 to each of multiple users U In step S210 described above, the instrument information acquisition unit 220 acquires the instrument information D1 generated by each of the multiple users U using the instrument information generation device 10, for each user U.

[0175] In other words, the instrument information acquisition unit 220 acquires instrument information D1, which is information generated based on the shape of the drawing figure P1 (figure) drawn by the user U's drawing operation, and which is associated with shape information D11 indicating the shape of the drawing figure P1 (figure), instrument type information D12 indicating the type of instrument based on the shape of the drawing figure P1 (figure), and frequency information D13 indicating the frequency assigned to each region of the drawing figure P1 (figure).

[0176] Furthermore, in step S220 described above, the display control unit 230 displays an instrument object P2 for each user U based on the instrument information D1 generated by each of the multiple users U.

[0177] In other words, the display control unit 230 superimposes an image based on the acquired instrument information D1, which is an instrument object P2 that is an image of a pattern corresponding to the shape of the drawing figure P1 (figure), and an operation part image P3 for each user U that is an image based on the detected posture and shows the operation part OP which is at least a part of the user U's body, and displays them on the display screen of the display device 51.

[0178] As described above, the type of instrument included in the instrument information D1 is selected based on the curvature of the lines that make up the drawn figure P1 (shape) and the complexity of the drawn figure P1 (shape), which is calculated based on the ratio of the distance from the start point to the end point of the lines that make up the drawn figure P1 (shape) to the length of the lines. The display control unit 230 displays instrument objects P2, each representing a different instrument type for each user U, and corresponding operation part images P3 for each user U. The contact area determination unit 250 determines the contact area RT for each user U based on the correspondence between the instrument object P2 and the operating area image P3 for each user U.

[0179] With the sound and video output system 12a configured in this way, multiple users U can perform performance operations simultaneously, thereby providing users U with greater enjoyment and satisfaction in the performance operation.

[0180] Furthermore, the sound and video output device 20 may output a different type of sound than when a single user U plays, if multiple users U play simultaneously.

[0181] For example, the sound output unit 260 may change at least one of the frequency and timbre depending on the number of detected users U.

[0182] Furthermore, the sound output unit 260 can output chords or control the progression of chords in response to the user U's performance operations. For example, when a single user U outputs chords, different chords may be output depending on the difference in the collision detection area between the operation part OP and the instrument object P2. Furthermore, for example, the sound output unit 260 may output chords of typical chord progressions that are considered pleasing in music theory (for example, repetitions of tonic and dominant chords, or cadence resolution progressions such as dominant-seventh chord → tonic chord). Here, the sound output unit 260 may produce richer chord structures and chord progressions when multiple users U are simultaneously performing the performance compared to when a single user U is performing the performance. For example, when multiple users U output chords, the output chords may have a richer sound (for example, chords with added tension notes such as 7ths and 9ths) compared to the chords output when a single user U is performing the performance.

[0183] In other words, the sound output unit 260 changes at least one of the following depending on the number of detected users U: the composition of the chords output simultaneously, or the progression of the chords output sequentially over time.

[0184] With the sound and video output system 12a configured in this way, when multiple users U perform the performance simultaneously, a richer musical expression is possible compared to when a single user U performs. Therefore, the sound and video output system 12a can provide users U with greater enjoyment and satisfaction compared to when a single user U performs.

[0185] Furthermore, when multiple users U perform performance operations simultaneously, the display mode of the operation part image P3 (also called the pointer image) displayed at the operation position PT may be made different for each user U. For example, the display control unit 230 may make the color and shape of the operation part image P3 different for each user U.

[0186] In other words, the display control unit 230 displays the pointer image on the display screen of the display device 51 in a manner that differs for each of the detected users U, indicating the operation area OP on the display screen of the display device 51.

[0187] With the sound and video output system 12a configured in this way, when multiple users U perform performance operations simultaneously, it is possible to visually represent which user U is operating the image P3 of the control part, making it easier to perform performance operations.

[0188] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and design modifications and the like are also included within the scope of the gist of the present invention. For example, a computer program to implement the functions of each of the above-mentioned devices may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. The term "computer system" here may include hardware such as an operating system and peripheral devices.

[0189] Furthermore, "computer-readable recording media" refers to writable non-volatile memory such as flexible disks, magneto-optical disks, ROMs, and flash memory, portable media such as DVDs (Digital Versatile Discs), and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" also includes volatile memory (such as DRAM (Dynamic Random Access Memory)) within computer systems that act as servers or clients when programs are transmitted via networks such as the Internet or communication lines such as telephone lines, which retain programs for a certain period of time.

[0190] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" used to transmit the program refers to a medium that has the function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line. Furthermore, the above program may be intended to implement some of the functions described above. It may also be a so-called differential file (differential program) that can implement the aforementioned functions in combination with programs already recorded in the computer system. [Explanation of Symbols]

[0191] 1…Performance system, 11…Instrument information generation system, 12, 12a…Audio and video output system

Claims

1. A detection unit that non-contactually detects the posture of multiple users, An instrument information acquisition unit acquires instrument information that is generated based on the shape of a figure drawn by a user's drawing operation, and which associates shape information indicating the shape of the figure, instrument type information indicating the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each region of the figure. A display control unit that superimposes and displays on a display screen an image of an instrument object, which is an image of a pattern corresponding to the shape of the figure, based on the acquired instrument information, and an image of a user-specific operating part, which is an image of an operating part that is at least a part of the user's body, based on the detected posture, Based on the detected posture, a coordinate calculation unit calculates the coordinates of the operation part image displayed on the display screen as operation coordinate information for each user, A contact area determination unit determines, for each user, a contact area among a plurality of areas of the instrument object displayed on the display screen that the operation position on the display screen indicated by the operation coordinate information has come into contact with, based on the operation coordinate information and the shape information of the instrument object. A sound output unit that outputs a sound corresponding to the frequency assigned to the contact area and the timbre of the instrument type indicated by the instrument information for each contact area determined for each user, A sound and video output device equipped with the following features.

2. The contact area determination unit determines, for each of the multiple operation positions indicated by the operation coordinate information of each of the multiple users that come into contact with a single instrument object displayed on the display screen, The sound output unit outputs sounds corresponding to each of the multiple contact areas determined for each user for a single instrument object. The sound and video output device according to claim 1.

3. The type of instrument included in the instrument information is selected based on the curvature of the lines constituting the drawn figure and the complexity of the figure calculated based on the ratio of the distance from the start point to the end point of the lines constituting the figure to the length of the lines. The display control unit displays the instrument objects, each of which is different for each user, in association with the operation part images for each user. The contact area determination unit determines the contact area for each user based on the correspondence between the instrument object and the operation area image for each user. The sound and video output device according to claim 1.

4. The sound output unit changes at least one of the frequency and the timbre according to the number of detected users. The sound and video output device according to claim 1.

5. On the computer, The ability to detect the posture of multiple users individually without physical contact, To obtain instrument information that is generated based on the shape of a figure drawn by the user's drawing operation, and which associates shape information indicating the shape of the figure, the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each region of the figure; The system displays on a display screen an image superimposed with an image of a musical instrument object, which is an image of a pattern corresponding to the shape of the graphic, and an image of a user-specific operating part, which is an image of an operating part that is at least a part of the user's body, based on the detected posture. Based on the detected posture, the coordinates of the image of the operating part displayed on the display screen are calculated for each user as operating coordinate information. Based on the aforementioned operation coordinate information and the shape information of the instrument object, the contact area is determined for each user, which is one of the multiple areas of the instrument object displayed on the display screen that the operation position on the display screen indicated by the operation coordinate information has come into contact with. For each contact area determined for each user, a sound is output that has a frequency assigned to the contact area and a timbre corresponding to the type of instrument indicated by the instrument information. A program to execute.

Images