Audio / video output device and program

Abstract

An audio / video output device according to the present invention includes a detection unit that detects a posture of a user in a non-contact manner, a musical instrument information acquisition unit that acquires musical instrument information in which shape information indicating a shape of a figure drawn by a drawing operation of the user, musical instrument type information indicating a type of a musical instrument based on the shape of the figure, and frequency information indicating a frequency assigned to each region, are associated, a display control unit that superimposes a musical instrument object and an operating portion image and performs display thereof on a display screen, a coordinate calculation unit that calculates coordinates on the display image of the operating portion image displayed on the display screen, as operating coordinate information, on the basis of the detected posture, a contact region determination unit that determines a contact region on the basis of the operating coordinate information and the shape information of the musical instrument object, and a sound output unit that outputs a sound that is of a frequency assigned to the determined contact region and is of a tone corresponding to the type of the musical instrument indicated by the musical instrument information.

Description

Audio-visual output device and program

[0001] The present invention relates to an audio-visual output device and a program. This application claims priority based on Japanese Patent Application No. 2024-163617 and Japanese Patent Application No. 2024-164338, filed in Japan on September 20, 2024, and incorporates the contents thereof herein.

[0002] For example, as an example of an interactive music installation, research on an experiential installation that presents an association between a visual impression and a timbre has been carried out. (See, for example, Non-Patent Document 1.)

[0003] Proceedings of the 27th Symposium of the Information Processing Society of Japan (INTERACTION 2023) (1C-53) Proposal for an Interactive Music Experience by Timbre Generation Based on the Visual Impression of Shapes, Wataru Hayashida, Shigenori Mochizuki (Ritsumeikan University)

[0004] However, in the above-described conventional experiential installation, although a method of associating a visual impression with a timbre has been presented, no proposal has been made regarding the configuration of a specific application. By configuring a specific application, it is preferable to provide a user with a performance experience that can obtain greater enjoyment and satisfaction.

[0005] An object of the present invention is to provide an audio-visual output device and a program that can provide a user with a performance experience that can obtain greater enjoyment and satisfaction.

[0006] One aspect of this embodiment includes a detection unit that non-contactually detects the user's posture, an instrument information acquisition unit that acquires instrument information, which is generated based on the shape of a figure drawn by the user's drawing operation and associates shape information indicating the shape of the figure, instrument type information indicating the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each of the regions into which the figure is divided, and an instrument object which is an image based on the acquired instrument information, and an operation part image which is an image based on the detected posture and indicates an operation part that is at least a part of the user's body, and displays these superimposed on a display screen. The sound-video output device comprises: a display control unit that displays; a coordinate calculation unit that calculates the coordinates on the display screen of the operation part image displayed on the display screen as operation coordinate information based on the detected posture; a contact area determination unit that determines, based on the operation coordinate information and the shape information of the instrument object, which of the multiple areas of the instrument object displayed on the display screen is the area that the operation position on the display screen indicated by the operation coordinate information has contacted; and a sound output unit that outputs a sound with a frequency assigned to the determined contact area and a tone corresponding to the type of instrument indicated by the instrument information.

[0007] One aspect of this embodiment involves a computer that performs a detection step to non-contactually detect the user's posture; an instrument information acquisition step to acquire instrument information which is generated based on the shape of a figure drawn by the user's drawing operation, and which associates shape information indicating the shape of the figure, the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each of the regions into which the figure is divided; and an image based on the acquired instrument information which is an instrument object, which is an image of a pattern corresponding to the shape of the figure, and an operating part image, which is an image based on the detected posture, which is an image showing an operating part that is at least a part of the user's body, and displays This program causes the execution of the following steps: a display control step to display on a display screen; a coordinate calculation step to calculate the coordinates on the display screen of the operation part image displayed on the display screen as operation coordinate information based on the detected posture; a contact area determination step to determine, based on the operation coordinate information and the shape information of the instrument object, that the contact area is the area on the display screen that the operation position on the display screen indicated by the operation coordinate information has contacted among a plurality of areas of the instrument object displayed on the display screen; and a sound output step to output a sound that has a frequency assigned to the determined contact area and has a timbre corresponding to the type of instrument indicated by the instrument information.

[0008] According to this embodiment, it is possible to provide users with a performance experience that offers greater enjoyment and satisfaction.

[0009] This figure shows an example of the device configuration of the performance system of this embodiment. This figure shows an example of the functional configuration of the instrument information generation system of this embodiment. This figure shows an example of the operation flow of the instrument information generation system of this embodiment. This figure shows an example of the correspondence between the drawing trajectory and the drawn figure of this embodiment. This figure shows an example of the determination space of the drawn figure of this embodiment. This figure shows an example of the correspondence between the shape of the drawn figure and the type of instrument of this embodiment. This figure shows an example of the correspondence between the drawn figure and the instrument object of this embodiment. This figure shows an example of the object attached to the instrument object of this embodiment. This figure shows an example of the correspondence between the shape of the drawn figure and the object attached to the instrument object of this embodiment. This figure shows an example of the correspondence between the area of ​​the drawn figure and the musical scale of this embodiment. This figure shows an example of the correspondence between the area of ​​the drawn figure and the musical scale of this embodiment. This figure shows an example of the instrument information of this embodiment. This figure shows an example of the functional configuration of the sound and video output system of this embodiment. This figure shows an example of the operation flow of the sound and video output system of this embodiment. This figure shows an example of the situation during performance of this embodiment. This figure shows an example of the animation operation of the instrument object of this embodiment. This figure shows an example of the display of the instrument object and the operation part image of this embodiment. This figure shows a modified example of the situation during performance of this embodiment.

[0010] The performance system 1 of this embodiment will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below. In all the figures used to describe the embodiments, components having the same function will be given the same reference numerals, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on another element in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX after calculations or processing have been performed on it. "XX" is any element (for example, any information).

[0011] [Embodiments] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Figure 1 is a diagram showing an example of the device configuration of the performance system 1 of this embodiment. The performance system 1 comprises a musical instrument information generation device 10, an audio-visual output device 20, a storage unit 30, a terminal device 40, and a display system 50.

[0012] The instrument information generation device 10 and the sound and video output device 20 are connected to the storage unit 30 by network N1, enabling mutual exchange of information. The instrument information generation device 10 and the sound and video output device 20 are connected to the terminal device 40 by network N2, enabling mutual exchange of information. The instrument information generation device 10 and the sound and video output device 20 are connected to the display system 50 by network N3, enabling mutual exchange of information. Networks N1 to N3 can exchange information with each other via wireless or wired communication. The following explanation will omit details regarding the exchange of information between each device via networks N1 to N3.

[0013] The instrument information generation device 10 and the sound and video output device 20 are computer devices such as personal computers. In this embodiment, the instrument information generation device 10 and the sound and video output device 20 are described separately as separate computer devices, but this is not limited to them; the instrument information generation device 10 and the sound and video output device 20 may be implemented by a single computer device.

[0014] The storage unit 30 is composed of, for example, a hard disk drive or semiconductor memory (flash memory, RAM, ROM), and stores various types of information, such as programs and data read by the instrument information generation device 10 and the sound and video output device 20. The storage unit 30 may also be implemented by a virtual storage device such as a cloud server located outside the instrument information generation device 10 and the sound and video output device 20.

[0015] The terminal device 40 is a computer device such as a smartphone, tablet, or personal computer, and includes an operation detection unit 41 and a terminal display unit 42. The operation detection unit 41 includes an operation device such as a touch panel, mouse, or keyboard, and detects user U's operations. The terminal display unit 42 includes a display device such as a liquid crystal display, and presents various images to user U. In one example of this embodiment, the terminal device 40 is configured as a so-called drawing tablet device and has the function of transmitting a drawing figure P1 drawn by user U to the musical instrument information generation device 10.

[0016] The display system 50 includes a display device 51, an imaging unit 52, and a sound output unit 53. The display device 51, for example, is equipped with a liquid crystal display and displays various images based on image information output by the instrument information generation device 10 and the sound and video output device 20. In one example of this embodiment, the display device 51 consists of two displays, a first display device 511 and a second display device 512, arranged side by side. The number of display devices 51 may be one or three or more.

[0017] The imaging unit 52, for example, is equipped with a video camera and captures images of the area within the detection area 60 in front of the display device 51. The imaging unit 52 transmits the captured video images to the sound and video output device 20.

[0018] The sound output unit 53 is equipped with, for example, an audio device such as an amplifier or a speaker, and outputs sound based on sound information transmitted from the sound-video output device 20.

[0019] In the following description, the combination of the instrument information generation device 10, the storage unit 30, and the terminal device 40 is also referred to as the instrument information generation system 11. Furthermore, the combination of the sound and video output device 20, the storage unit 30, and the display system 50 is also referred to as the sound and video output system 12. Note that the instrument information generation system 11 may include the display system 50, and the sound and video output system 12 may include the terminal device 40.

[0020] The instrument information generation system 11 and the sound and video output system 12 will be described below, in that order. First, the functional configuration and operation flow of the instrument information generation system 11 will be explained with reference to Figures 2 and 3.

[0021] [About the Instrument Information Generation System] Figure 2 is a diagram showing an example of the functional configuration of the instrument information generation system 11 of this embodiment. Figure 3 is a diagram showing an example of the operation flow of the instrument information generation system 11 of this embodiment.

[0022] The musical instrument information generation device 10 includes a calculation unit 100. The calculation unit 100 includes, for example, a central processing unit (CPU), and operates based on programs and data stored in the storage unit 30, providing various functions.

[0023] The calculation unit 100 includes, as its functional units, a drawing operation reception unit 110, a type determination unit 120, a tone adjustment unit 130, a frequency assignment unit 140, a musical instrument information generation unit 150, and an object addition unit 160.

[0024] (Step S110) User U draws on the operation detection unit 41 (for example, a touch panel) of the terminal device 40. Drawing means that User U draws any shape. The operation of drawing a shape on the operation detection unit 41 is also called a drawing operation.

[0025] Figure 4 shows an example of the correspondence between the drawing trajectory OT1 and the drawn shape P1 in this embodiment. User U draws the drawing trajectory OT1 shown in the figure on the operation detection unit 41 (for example, a touch panel) using a stylus or finger. The operation detection unit 41 obtains the coordinates of the drawing trajectory OT1 on the touch panel. The terminal display unit 42 displays the drawn shape P1 corresponding to the drawing trajectory OT1 detected by the operation detection unit 41.

[0026] In one example of this embodiment, the performance system 1 targets kindergarten or nursery school children, or elementary school students. Users U of this target group can quickly draw familiar shapes if they are single-stroke drawings. The instrument information generation device 10 of this embodiment acquires the single-stroke trajectory as the drawing trajectory OT1. In this example, user U draws a heart-shaped drawing trajectory OT1. In this case, the terminal display unit 42 displays the first drawing shape P11 (for example, a heart-shaped shape) shown in the figure. The heart-shaped shape is an example of a single-stroke drawing shape.

[0027] The terminal device 40 transmits the coordinate information of the drawing trajectory OT1 detected by the operation detection unit 41 to the instrument information generation device 10. The drawing operation reception unit 110 of the instrument information generation device 10 acquires the coordinate information of the drawing trajectory OT1 transmitted from the terminal device 40. The acquisition of the coordinate information of the drawing trajectory OT1 can also be described as the drawing operation reception unit 110 accepting a drawing operation from user U.

[0028] In other words, the drawing operation reception unit 110 receives drawing operations from user U.

[0029] As shown in the figure, the coordinates of the drawing trajectory OT1 and the drawn figure P1 are approximately the same. In the following explanation, when the drawing trajectory OT1 and the drawn figure P1 are not distinguished, they will be collectively referred to as the drawn figure P1 drawn by user U (or simply as the drawn figure P1).

[0030] [Determination of the type of instrument] (Step S120) Returning to Figure 3, the type determination unit 120 determines the type of instrument based on the drawing shape P1 drawn by the user U. The correspondence between the drawing shape P1 and the type of instrument will be explained below.

[0031] Figure 5 shows an example of the determination space for the drawn figure P1 in this embodiment. The determination space for the drawn figure P1 is composed of coordinate axes that classify the shape of the drawn figure P1 (in this example, a figure drawn in a single stroke) drawn by the user U into multiple types.

[0032] In one example of this embodiment, the judgment space of the drawn figure P1 is composed of two coordinate axes: the curvature axis AX1 and the complexity axis AX2.

[0033] The curvature axis AX1 is an axis that indicates whether the curvature of the lines constituting the drawn figure P1 is large (more curved components) or small (fewer curved components and closer to a straight line). The curvature of the lines constituting the drawn figure P1 can be calculated by various algorithms. For example, the type determination unit 120 divides the trajectory of the lines constituting the drawn figure P1 (i.e., the drawing trajectory OT1) into small intervals, and calculates the curvature of the lines constituting the drawn figure P1 by accumulating (or integrating) the curvature of each small interval along the trajectory.

[0034] In the following explanation, the curvature of the lines that make up the drawn figure P1 will also simply be referred to as the curvature of the drawn figure P1.

[0035] The complexity axis AX2 indicates whether the complexity of the lines constituting the drawn figure P1 is large (more complex shape) or small (simpler shape). The complexity of the lines constituting the drawn figure P1 can be calculated using various algorithms. For example, the type determination unit 120 calculates the complexity of the lines constituting the drawn figure P1 based on the ratio of the distance from the start point to the end point of the lines constituting the drawn figure P1 to the length of the lines constituting the drawn figure P1.

[0036] Generally, the ratio of the distance from the start point to the end point of a line to the length of the line can also be said to be the ratio of the shortest distance between the start and end points to the length of the trajectory when moving along the line (i.e., the distance traveled). If the shortest distance and the distance traveled for a given line are approximately equal, then the line can be said to represent a trajectory that moves along the shortest route, and is considered to have the simplest shape. Conversely, if the shortest distance and the distance traveled for a given line are different, then the line can be said to represent a trajectory that moves along a route that is not the shortest route, and is considered to have a complex shape. Furthermore, if the ratio of the shortest distance to the distance traveled (where the shortest distance is the denominator and the distance traveled is the numerator) is large, then the line can be said to have a more complex shape. In other words, the ratio (or proportion) of the shortest distance to the distance traveled for a given line indicates the complexity of that line.

[0037] In other words, the type determination unit 120 determines the type of musical instrument based on the curvature of the lines that make up the drawn figure P1 (figure) and the complexity of the drawn figure P1 (figure), which is calculated based on the ratio of the distance from the start point to the end point of the lines that make up the drawn figure P1 (figure) to the length of the lines.

[0038] In the following explanation, the complexity of the lines that make up the drawn figure P1 will also simply be referred to as the complexity of the drawn figure P1.

[0039] [Complexity Calculation Algorithm (Variation)] The complexity calculation algorithm described above is just one example and is not limited to it. For example, complexity may be calculated as follows.

[0040] (Modification 1) (1) Calculate the rectangular frame (outer frame) that circumscribes the drawn figure P1. (2) Calculate the length of the circle (or arc) inscribed in the outer frame as the reference length. For example, if the outer frame is a square, calculate the length (circumference) of the circle inscribed in the outer frame and use it as the reference length. For example, if the outer frame is a rectangle, calculate the length of the arc inscribed in the outer frame and use it as the reference length. (3) Compare the reference length calculated in (2) above with the length of the drawing trajectory OT1 of the drawn figure P1. A small ratio of the length of the drawing trajectory OT1 to the reference length indicates low complexity, and a large ratio indicates high complexity. Calculate the complexity as the ratio of the reference length to the length of the drawing trajectory OT1 of the drawn figure P1.

[0041] (Modification 2) (1) Calculate the perimeter (sum of the lengths of each side) of the rectangular frame (outer frame) that circumscribes the drawn figure P1. (2) Use the perimeter calculated in (1) above as the base length and compare it with the length of the drawing trajectory OT1 of the drawn figure P1. A small ratio of the length of the drawing trajectory OT1 to the base length indicates low complexity, while a large ratio indicates high complexity. The ratio of the base length to the length of the drawing trajectory OT1 of the drawn figure P1 is calculated as the complexity.

[0042] Incidentally, in the above-described Modification 2, based on the curvature of the drawing locus OT1 of the drawing figure P1, the length obtained by multiplying the perimeter by a predetermined magnification may be used as the reference length. For example, when the curvature of the drawing locus OT1 is small (there are many straight portions), the length obtained by multiplying the perimeter by 1 / 2 is used as the reference length. When the curvature of the drawing locus OT1 is large (there are many curved portions), the perimeter is used as the reference length.

[0043] Also, among the complexities calculated in parallel by the above-described respective calculation algorithms, the maximum (or minimum) complexity may be adopted as the complexity of the drawing figure P1.

[0044] [Selection of Musical Instrument Type] In an example of this embodiment, the type determination unit 120 selects the type of musical instrument from among a plurality of types of musical instruments stored in advance in association with the curvature and complexity.

[0045] More specifically, the type determination unit 120 calculates the curvature and complexity of the drawing figure P1. The type determination unit 120 determines at which coordinates in the determination space indicated by the curvature axis AX1 and the complexity axis AX2 the calculated curvature and complexity of the drawing figure P1 are located. The type determination unit 120 selects the type of musical instrument associated with the coordinates at which the curvature and complexity of the drawing figure P1 are located. An example of the correspondence relationship between the shape of the drawing figure P1 and the type of musical instrument will be described with reference to FIG. 6.

[0046] FIG. 6 is a diagram showing an example of the correspondence relationship between the shape of the drawing figure P1 and the type of musical instrument in this embodiment. The type determination unit 120 selects a musical instrument corresponding to the coordinates of the shape (for example, curvature and complexity) of the drawing figure P1 from among the plurality of musical instruments shown in the figure.

[0047] Generally, as an example of "sound symbolism" in which sound is an event that evokes a certain specific image, in the field of psychology, the so-called "bouba / kiki" effect has been pointed out. This also means that there is a correspondence relationship between the visual impression of a figure and the aspect of sound (for example, frequency, harmonic composition, degree of reverberation, volume, etc.). That is, the visual impression of a figure changes, for example, with the curvature and complexity of the figure as parameters.

[0048] For example, depending on the type of musical instrument, aspects of the sound such as the frequency output when played, the harmonic components, and the envelope curve indicating the temporal change in the sound intensity vary widely. Due to these differences in the aspects of the sound, the shape of the figure evoked in user U by sound symbolism varies. As an example, as shown in the same figure, in the case of the drawing figure P1 with a small curvature and a straight line, and a low complexity and simplicity, a glockenspiel that produces a sharp, straight, and relatively simple sound is imagined. The type determination unit 120 selects a musical instrument that produces the sound symbolized by the drawing figure P1 drawn by user U based on the above-described theory of sound symbolism.

[0049] That is, the type determination unit 120 determines the type of musical instrument based on the shape of the drawing figure P1 (figure) drawn by the drawing operation.

[0050] [Association of Musical Scale with Musical Instrument Object] (Step S130) Returning to FIG. 3, the musical instrument information generation unit 150 generates the shape of the musical instrument object P2 based on the drawing figure P1 drawn by user U.

[0051] FIG. 7 is a diagram showing an example of the correspondence relationship between the drawing figure P1 and the musical instrument object P2 of the present embodiment. The figure shows an example of the correspondence relationship between the first drawing figure P11 to the fifth drawing figure P15 and the first musical instrument object P21 to the fifth musical instrument object P25.

[0052] The musical instrument information generation unit 150 generates the shape of the musical instrument object P2 by making the shape of the musical instrument object P2 and the shape of the drawing trajectory OT1 (for example, a single-stroke line) of the drawing figure P1 similar. According to the musical instrument information generation device 10 configured in this way, user U can recognize that the musical instrument object P2 is not given by the computer device but is created by himself / herself. As a result, the sense of ownership and familiarity of user U with respect to the musical instrument object P2 can be enhanced.

[0053] Furthermore, User U's sense of ownership and familiarity with the instrument object P2 refers to User U's desire to possess the instrument object P2, similar to the feelings of "cute" or "fun to be with" one has towards a pet they own. In the following explanation, these feelings of User U towards the instrument object P2 will be collectively referred to as "liking for the instrument object P2" (or simply "liking").

[0054] (Step S140) Returning to Figure 3, the object attachment unit 160 attaches predetermined objects to the instrument object P2. These predetermined objects include an eye object OBJ1 and a mouth object OBJ2. In the following description, the predetermined objects that the object attachment unit 160 attaches to the instrument object P2 are also referred to as attached objects.

[0055] Figure 8 shows an example of an object attached to the instrument object P2 in this embodiment. By attaching the eye object OBJ1 and the mouth object OBJ2 to the instrument object P2, the instrument object P2 is personified. As a result, the user U's sense of ownership and familiarity with the instrument object P2 can be enhanced.

[0056] The added objects (for example, the eye object OBJ1 and the mouth object OBJ2) are designed with shapes and positions that enhance the user's favorability towards the instrument object P2. An example of an algorithm for placing added objects is described below.

[0057] The object attachment unit 160 sets a reference position for the drawing figure P1. Since the drawing figure P1 and the instrument object P2 are similar in shape, it can also be said that "the object attachment unit 160 sets a reference position for the instrument object P2." Similarly, in the following explanation, the part referred to as the drawing figure P1 can be replaced with the instrument object P2.

[0058] The reference position is preferably a position that can be geometrically derived from various shapes of drawn figures P1. For example, the reference position is the centroid of the area enclosed by the drawing trajectory OT1 of the drawn figure P1. In this case, the object attachment unit 160 sets the centroid of the area of ​​the drawn figure P1 (centroid CG1 as exemplified in the figure) as the reference position.

[0059] The object attachment unit 160 places the attached object on the circumference of a virtual reference circle (reference circle CL1 as illustrated in the figure) with radius r centered on a reference position (for example, the centroid CG1).

[0060] In other words, the object attachment unit 160 attaches the eye object OBJ1 and the mouth object OBJ2 at positions based on the reference position of the drawing shape P1 (shape).

[0061] With the instrument information generation device 10 configured in this way, even if the shapes of the drawn figures P1 are all different, additional objects can be placed based on a predetermined placement algorithm.

[0062] Note that for the eye object OBJ1 and the mouth object OBJ2, objects with shapes corresponding to the shape of the drawing figure P1 may be selected.

[0063] Figure 9 shows an example of the correspondence between the shape of the drawing figure P1 in this embodiment and the object attached to the instrument object P2. The figure shows an example where the shape of the attached object is associated with the position of the shape of the drawing figure P1 on the curvature axis AX1.

[0064] The closer the position of the drawing shape P1 on the curvature axis AX1 is to a straight line, the harder (sharper) the outlines of the eye object OBJ1 and mouth object OBJ2 will be. The closer it is to a curve, the softer (rounder) the outlines of the eye object OBJ1 and mouth object OBJ2 will be.

[0065] The added objects may also be displayed with animation. Figures [A] and [B] show examples of how added objects can be displayed with animation. For example, the eye object OBJ1 can be displayed with animation such as blinking and moving the pupils. The mouth object OBJ2 can be displayed with animation such as opening and closing the mouth.

[0066] Furthermore, the shape of the added object may be associated with the complexity axis AX2. Alternatively, the shape of the added object may be associated with a combination of the curvature axis AX1 and the complexity axis AX2. In other words, the shape of the added object is determined based on at least one of the curvature axis AX1 and the complexity axis AX2.

[0067] In other words, the object attachment unit 160 attaches the eye object OBJ1 representing the character's eyes and the mouth object OBJ2 representing the character's mouth to the drawing figure P1 (shape) at a position based on the reference position of the drawing figure P1 (shape) by selecting the pattern of the eye object OBJ1 representing the character's eyes and the pattern of the mouth object OBJ2 representing the character's mouth, based on at least one of the curvature of the lines constituting the drawing figure P1 (shape) drawn by the drawing operation and the complexity of the drawing figure P1 (shape) calculated based on the ratio of the distance from the start point to the end point of the lines constituting the drawing figure P1 (shape) to the length of the lines.

[0068] Note that the shape of the added object being associated with the curvature axis AX1 or the complexity axis AX2 is just one example, and is not limited to this. The shape of the added object may be determined based on some other determination axis other than the curvature axis AX1 or the complexity axis AX2 corresponding to the shape of the drawing figure P1.

[0069] Returning to Figure 3, the frequency assignment unit 140 assigns frequencies corresponding to musical scales to the shape of the instrument object P2 generated by the instrument information generation unit 150. A specific example of the frequency assignment procedure by the frequency assignment unit 140 will now be described.

[0070] (Step S150) The frequency allocation unit 140 divides the drawing figure P1 into multiple regions R.

[0071] Figure 10 shows an example of the correspondence between the region R of the drawing figure P1 in this embodiment and a musical scale. In this example, the case where the drawing figure P1 is the first drawing figure P11 (for example, a heart-shaped figure) is shown.

[0072] The frequency allocation unit 140 unfolds the outline of the drawn figure P1 (figure) in a predetermined direction and divides the unfolded drawn figure P1a.

[0073] In the example shown in the figure, the frequency allocation unit 140 unfolds the drawn figure P1 in the Xp1 axis direction and divides the unfolded drawn figure P1a into eight parts in the Xp1 axis direction. When the drawn figure P1a is divided in the Xp1 axis direction, eight regions R1 to R8 are formed.

[0074] (Step S160) The frequency assignment unit 140 assigns frequencies according to the musical scale for each region R.

[0075] In other words, the frequency assignment unit 140 assigns a frequency corresponding to the musical scale to each region R into which the drawn figure P1 (figure) is divided.

[0076] The frequency of the note A4, according to the so-called international musical notation, is 440.000 Hz. Similarly, the frequency of the note C4 is 261.626 Hz. The frequency of the note C5 is 523.251 Hz. Note that these frequencies are just examples when the reference tone (A4) is set at 440.000 Hz, and for a typical musical scale (a frequency obtained by dividing an octave into 12 notes), the frequencies may shift to higher or lower frequencies.

[0077] In this example, the frequency assignment unit 140 assigns the frequency of sound C4 (261.626 [Hz]) to region R1. Similarly, the frequency assignment unit 140 assigns the frequencies of sounds D4 to C5 to regions R2 to R8, respectively.

[0078] In other words, the frequency assignment unit 140 unfolds the outline of the drawn figure P1 (figure) in a predetermined direction and assigns musical scale frequencies to the regions divided from the unfolded drawn figure P1a.

[0079] Here, the frequency assignment unit 140 assigns the so-called white key scale to region R, but does not assign the black key scale. In other words, the frequency assignment unit 140 assigns only the white key scale to region R. In the following explanation, the scale composed of the white and black keys of a piano is also called a diatonic scale (12-tone scale). Among the notes that make up the diatonic scale, the notes corresponding to the white keys of a piano are also called the diatonic group KW, and the notes corresponding to the black keys of a piano are also called the derived group KB.

[0080] In other words, the frequency assignment unit 140 assigns the frequencies of eight consecutive notes included in the diatonic group KW, which constitutes the diatonic scale, to the region R.

[0081] Generally, when sounds from the derived sound group KB are pronounced simultaneously with other sounds, the resulting sound can feel unnatural (for example, like a dissonance), compared to when multiple sounds from the diatonic sound group KW are pronounced simultaneously. In this embodiment, the frequency assignment unit 140 excludes the derived sound group KB from the frequencies assigned to the region R, thereby reducing the unnatural feeling when multiple frequencies are pronounced simultaneously.

[0082] [Variations of frequency allocation]

[0083] (1) Variations of the range of scale assignment to region R In the example described above, note C4 is assigned to region R1, where the lowest frequency is assigned, and note C5 is assigned to region R8, where the highest frequency is assigned. In other words, the eight consecutive notes from region R1 to region R8 constitute the so-called C major scale, but this is not limited to this. Depending on the type of instrument, the sound may sound unnatural if the C major scale is used due to differences in the range of playable notes. Therefore, the frequency assignment unit 140 assigns the frequencies of the scale within the range in which the sound is natural according to the type of instrument to region R. For example, the frequency assignment unit 140 may assign the eight notes from note A3 to note A4 to region R1 to region R8. In this case, the eight consecutive notes from region R1 to region R8 constitute the so-called A natural minor scale in the natural minor scale.

[0084] (2) Modified Figure 11 of the shape of region R is a diagram showing an example of the correspondence between region R of the drawing figure P1 of this embodiment and musical scales. In the example shown in the diagram, the case in which the drawing figure P1 is a first drawing figure P11 (for example, a heart-shaped figure) is shown. In this modified case, the frequency assignment unit 140 divides the drawing figure P1 in a predetermined direction without unfolding the drawing figure P1 (that is, while keeping the shape of the drawing figure P1).

[0085] The frequency allocation unit 140 divides the drawn figure P1 in a predetermined direction. In the example shown in the figure, the frequency allocation unit 140 divides the drawn figure P1 into eight parts in the Xp1 axis direction. When the drawn figure P1 is divided in the Xp1 axis direction, eight regions R1 to R8 are formed.

[0086] In other words, the frequency assignment unit 140 assigns frequencies of a musical scale to each of the eight regions R obtained by dividing the drawn figure P1 (figure) into eight sections in a predetermined direction. These regions are created by dividing the area between a reference tone corresponding to the type of instrument and a tone one octave away from the reference tone into eight notes. In this example, R is divided into eight regions R1 to R8, but the number of divisions is not limited to eight. The frequency assignment unit 140 assigns frequencies of a musical scale to each of the eight regions R divided into a number of sections corresponding to the number of notes included in a predetermined musical scale. These regions are created by dividing the area between a reference tone corresponding to the type of instrument and a tone one octave away from the reference tone according to the number of divisions. In other words, eight divisions are just one example of the number of divisions described above.

[0087] With the instrument information generation device 10 configured in this way, the instrument object P2 can be likened to instruments that express musical scales, such as keyboard instruments like pianos and clavinets, keyboard percussion instruments like xylophones and glockenspiels, wind instruments like flutes and oboes, and string instruments like violins and guitars. Therefore, the instrument information generation device 10 makes it possible to express sound effects produced by various playing techniques characteristic of these instruments (for example, glissando).

[0088] (3) Modifications of the direction of division of region R In the example described above, the direction of division of region R was explained as being the xp1 axis direction, but it is not limited to this. The direction of division of region R may also be the yp1 axis direction. Furthermore, region R may be divided in a matrix shape that combines the xp1 axis direction and the yp1 axis direction.

[0089] In either frequency assignment method, the frequency assignment unit 140 assigns only the fundamental tone group KW to region R, and does not assign the derived tone group KB to region R. Since the frequency assignment unit 140 in this embodiment excludes the derived tone group KB from the frequencies assigned to region R, it is possible to reduce the unnatural feeling when multiple frequencies are pronounced simultaneously.

[0090] (Step S170) Returning to Figure 3, the tone adjustment unit 130 adjusts the tone according to the type of instrument selected in step S120 based on various conditions. As an example, the tone adjustment unit 130 adjusts the tone according to the drawing speed of the drawing figure P1 by the user U. The drawing speed of the drawing figure P1 refers to the drawing time required to draw the drawing trajectory OT1 from the start point to the end point, or the ratio of the length of the line of the drawing trajectory OT1 from the start point to the end point to the drawing time.

[0091] In other words, the tone adjustment unit 130 adjusts the tone of the instrument based on the drawing speed of the drawing figure P1 (figure) created by the drawing operation.

[0092] For example, a short drawing time (i.e., when user U draws quickly) evokes a bright, lively, or hard tone. A long drawing time (i.e., when user U draws slowly and deliberately) evokes a rich, relaxed, or soft tone.

[0093] The tone adjustment unit 130 adjusts the tone of the instrument to a tone associated with the drawing speed described above, based on the drawing speed of the drawn figure P1. With the instrument information generation device 10 configured in this way, it is possible to provide the instrument object P2 with a tone that is in line with the user U's explicit drawing intention, or with sound symbols that appear in the drawn figure P1 without the user U being aware of them.

[0094] [Generation of instrument information] (Step S180) The instrument information generation unit 150 generates instrument information D1 by summarizing the parameters of the instrument object P2 generated in steps S120 to S170.

[0095] Figure 12 shows an example of instrument information D1 in this embodiment. Instrument information D1 consists of an instrument information ID that individually identifies the generated instrument object P2, and various parameters. The various parameters include the shape of the instrument object P2, the type of instrument, the position of region R within the instrument object P2, the frequency assigned to each region R, the result of timbre adjustment, the shape and position of the added object, and so on.

[0096] Shape information D11 is information indicating the shape of the instrument object P2. Instrument type information D12 is information indicating the type of instrument determined by the type determination unit 120. Frequency information D13 is information indicating the frequency assigned to region R by the frequency assignment unit 140. Timbre information D14 is information indicating the timbre adjusted by the timbre adjustment unit 130. Instrument information D1 is composed of these shape information D11, instrument type information D12, frequency information D13, and timbre information D14.

[0097] In other words, the instrument information generation unit 150 generates instrument information D1 by associating shape information D11, which indicates the shape of the drawn figure P1 (figure), instrument type information D12, which indicates the type of instrument that has been determined, and frequency information D13, which indicates the frequency assigned to the region R.

[0098] As described above, the instrument information D1 includes information on the tone adjusted by the tone adjustment unit 130 (tone information D14).

[0099] In other words, the instrument information generation unit 150 further associates the adjusted timbre information to generate instrument information D1.

[0100] Furthermore, as described above, the instrument information D1 may also include additional object information D15 that indicates the shape and position of the additional object.

[0101] In other words, the instrument information generation unit 150 further associates additional object information D15, which represents the eye object OBJ1 and the mouth object OBJ2, to generate instrument information D1.

[0102] (Step S190) Returning to Figure 3, the instrument information generation unit 150 stores the generated instrument information D1 in the storage unit 30, and completes the series of operations for generating the instrument information D1.

[0103] Next, the functional configuration and operation flow of the sound and video output system 12 will be explained with reference to Figures 13 and 14. The sound and video output system 12 is a system that outputs sound and video from an instrument object P2 using the instrument information D1 generated by the instrument information generation device 10 in the flow described above.

[0104] [About the Audio-Video Output System] Figure 13 is a diagram showing an example of the functional configuration of the audio-video output system 12 of this embodiment. Figure 14 is a diagram showing an example of the operation flow of the audio-video output system 12 of this embodiment. The audio-video output system 12 is composed of an audio-video output device 20, a storage unit 30, a display device 51, an imaging unit 52, and an audio output unit 53. The display device 51, imaging unit 52, and audio output unit 53 are collectively referred to as the display system 50.

[0105] In this embodiment, the sound and video output device 20 and the instrument information generation device 10 are described as separate computer devices, but this is not the only way to describe them. As mentioned above, the sound and video output device 20 and the instrument information generation device 10 may be implemented by a single computer device.

[0106] The memory unit 30 stores the instrument information D1 generated by the instrument information generation device 10. The configuration of the memory unit 30 is the same as that of the instrument information generation system 11, so its explanation is omitted.

[0107] The audio-video output device 20 is a computer device such as a personal computer. The audio-video output device 20 includes an arithmetic unit 200. The arithmetic unit 200 includes, for example, a central processing unit (CPU), and operates based on programs and data stored in the memory unit 30, providing various functions.

[0108] The calculation unit 200 includes, as its functional units, a detection unit 210, an instrument information acquisition unit 220, a display control unit 230, a coordinate calculation unit 240, a contact area determination unit 250, and a sound output unit 260.

[0109] (Step S210) The instrument information acquisition unit 220 acquires instrument information D1 from the storage unit 30. As described above, instrument information D1 is information generated based on the shape of the drawing figure P1 drawn by the user U's drawing operation. Instrument information D1 is information to which shape information D11, instrument type information D12, and frequency information D13 are associated with an instrument ID that identifies the instrument object P2.

[0110] In other words, the instrument information acquisition unit 220 acquires instrument information D1, which is information generated based on the shape of the drawing figure P1 (figure) drawn by the user U's drawing operation, and which is associated with shape information D11 indicating the shape of the drawing figure P1 (figure), instrument type information D12 indicating the type of instrument based on the shape of the drawing figure P1 (figure), and frequency information D13 indicating the frequency assigned to each region R of the drawing figure P1 (figure).

[0111] (Step S220) The display control unit 230 displays an image of the instrument object P2 based on the acquired instrument information D1 on the display device 51.

[0112] Figure 15 shows an example of the situation during performance in this embodiment. The display device 51 displays an image of the instrument object P2.

[0113] In the following description, the coordinate system on the display surface of the display device 51 is also referred to as the display coordinate system (display X coordinate xd, display Y coordinate yd). That is, the display control unit 230 displays the instrument object P2 at a predetermined position in the display coordinate system based on a predetermined display position calculation algorithm.

[0114] User U performs a gesture to operate the instrument object P2 within a detection area 60 set at a predetermined position in front of the display device 51. The gesture to operate the instrument object P2 is also called a performance operation.

[0115] The performance operation involves, for example, the user U moving their right or left hand towards the display device 51 within the detection area 60. In this case, the user U will be moving as if they were a conductor towards the display device 51. The detection area 60 can also be said to be the area to be captured by the imaging unit 52. In other words, the user U performs the performance operation within the area to be captured by the imaging unit 52. Next, a specific example of the display of the instrument object P2 by the display control unit 230 will be described.

[0116] [Animation of the instrument object P2] Figure 16 shows an example of the animation of the instrument object P2 in this embodiment. The display control unit 230 can animate the instrument object P2 based on a predetermined display algorithm.

[0117] (1) Animation display of added objects The display control unit 230 displays animations for the added objects. Added objects include, for example, eye object OBJ1 and mouth object OBJ2. The display control unit 230 makes the eye object OBJ1 blink and move the position of the pupil. The display control unit 230 also makes the mouth object OBJ2 open and close its mouth.

[0118] With the sound and video output device 20 configured in this way, the expressions of the instrument object P2 can be enriched, and the user U's favorability towards the instrument object P2 can be increased.

[0119] Furthermore, the display control unit 230 may display animations of the additional objects at a speed and frequency corresponding to the performance operation. With the sound and video output device 20 configured in this way, the instrument object P2 appears to be responding to the user U's performance operation, thereby enhancing the interactivity that user U perceives with the instrument object P2.

[0120] (2) Movement of the display position of the instrument object P2 The display control unit 230 may move the display position of the instrument object P2 on the display screen of the display device 51 in accordance with the passage of time. For example, the display control unit 230 moves the instrument object P2 along a movement trajectory OT2. The movement trajectory OT2 is, as an example, defined by the display range of the display device 51 as a frame, such that the angle of incidence and the angle of emission are the same (that is, according to the properties of reflection of a physical object in real space).

[0121] In other words, the display control unit 230 changes the position of the instrument object P2 on the display screen of the display device 51.

[0122] By having the instrument object P2 move along the movement trajectory OT2, the gameplay of playing the instrument is improved compared to when the instrument object P2 is stationary on the display screen, providing a greater sense of enjoyment for the user U.

[0123] Furthermore, the movement trajectory OT2 may be such that the angle of incidence and the angle of emission are different from each other (i.e., contrary to the properties of reflection of an object). If it is contrary to the properties of reflection of an object, the user U can be surprised by the movement trajectory OT2 of the instrument object P2, improving the game-like aspect of the performance operation and providing the user U with greater enjoyment.

[0124] [Detection of User U's Posture] (Step S230) Returning to Figure 14, the imaging unit 52 captures an image of User U within the detection area 60. The imaging unit 52 transmits the captured video image of User U to the sound and video output device 20.

[0125] The detection unit 210 of the sound and video output device 20 acquires a video image of user U received from the imaging unit 52. From the acquired video image, the detection unit 210 detects the user U's operating area OP. The operating area OP is a predetermined part of user U's body.

[0126] The detection of the user U's operating part OP by the detection unit 210 is also referred to as detecting the user U's posture. The detection unit 210 detects the user U's posture without using devices that come into contact with the user U and detect the user U's operations, such as a keyboard, mouse, joystick, handheld controller, or foot sensor.

[0127] In other words, the detection unit 210 detects the user U's posture without contact. Non-contact detection includes methods that use video of the user U captured by the imaging unit 52, as well as methods that use motion capture sensors such as infrared sensors and ultrasonic sensors.

[0128] In the following description, the coordinate system used by the detection unit 210 to detect the position of the operating part OP is also referred to as the real-space coordinate system (real-space X coordinate xr, real-space Y coordinate yr). The real-space coordinate system is a coordinate system that indicates the position within a plane set in the detection area 60. The real-space X coordinate xr is set parallel to the display X coordinate xd, and the real-space Y coordinate yr is set parallel to the display Y coordinate yd. In other words, the plane formed by the display coordinate system and the plane formed by the real-space coordinate system are set parallel to each other. Note that parallelism here does not necessarily have to be mathematically parallel; it is sufficient that the position of the user U's operating part OP in the real-space coordinate system can be converted to the display position on the display device 51 in the display coordinate system.

[0129] (1) In the case of one operating area OP, the operating area OP may be a specific single part of the user U's body. For example, the operating area OP may be only the fingertips of the user U's right hand (first operating area OP1).

[0130] (2) When there are multiple operating areas OP, the operating area OP does not necessarily have to be one specific location. The operating area OP may be multiple parts of the user U's body. For example, as shown in Figure 15, the operating area OP may be the fingertips of the user U's right hand (first operating area OP1) or the fingertips of the left hand (second operating area OP2).

[0131] In other words, the operating area OP includes a first operating area OP1 and a second operating area OP2, each corresponding to a different part of the user U's body.

[0132] In the example described above, the detection unit 210 is described as detecting a predetermined part (for example, a fingertip) as the operating part OP, but it is not limited to this. For example, the detection unit 210 may be able to detect the speed of movement of an object in the detection area 60. In this case, the detection unit 210 may detect the part of the user U's body that is moving the fastest as the operating part OP. Also, the detection unit 210 may be able to detect the position in the depth direction in the detection area 60. In this case, the detection unit 210 may detect the part of the user U's body that is closest to the display device 51 (or imaging unit 52) ​​as the operating part OP.

[0133] [Display of Operation Area Images] Figure 17 shows an example of the display of the instrument object P2 and the operation area image P3 in this embodiment. The operation area image P3 is an image showing the position of the operation area OP on the display screen of the display device 51. The figure shows the positional relationship between two types of operation positions PT (operation position PT1, operation position PT2) and the instrument object P2 on the display screen of the display device 51. The first operation area image P31 shown in the figure is the operation area image P3 showing operation position PT1. The second operation area image P32 is the operation area image P3 showing operation position PT2.

[0134] (Step S240) Returning to Figure 14, the coordinate calculation unit 240 calculates the coordinates (display X coordinate xd, display Y coordinate yd) of the operation part image P3 to be displayed on the display screen of the display device 51, based on the posture of the user U detected by the detection unit 210. The coordinates of the operation part image P3 on the display screen of the display device 51 are also called operation coordinate information D2. In other words, the coordinate calculation unit 240 calculates the operation coordinate information D2 by converting the coordinates of the operation part OP in the real space coordinate system to the coordinates in the display coordinate system. That is, the coordinates of the operation part image P3 (operation coordinate information D2) are obtained by converting the coordinates of the operation part OP in the real space coordinate system to the coordinates in the display coordinate system.

[0135] In other words, the coordinate calculation unit 240 calculates the coordinates of the operation part image P3 displayed on the display screen of the display device 51 as operation coordinate information D2, based on the detected posture.

[0136] The display control unit 230 displays the operation part image P3 corresponding to the operation part OP detected by the detection unit 210 on the display device 51. At this time, the display control unit 230 displays the operation part image P3 and the instrument object P2 simultaneously on the display device 51. In other words, the display control unit 230 displays the instrument object P2 and the operation part image P3 superimposed on the display screen of the display device 51.

[0137] In other words, the display control unit 230 superimposes an instrument object P2, which is an image based on the acquired instrument information D1, and an operation part image P3, which is an image based on the detected posture and shows an operation part OP, which is at least a part of the user U's body, and displays them on the display screen of the display device 51.

[0138] In the following description, the position of the operating part OP on the display screen of the display device 51 is also referred to as the operating position PT. That is, the operating part image P3 is an image that shows the operating position PT. When the user U changes their posture (for example, when they move the operating part OP), the display control unit 230 changes the display position of the operating part image P3 in accordance with the change in posture.

[0139] Up to this point, the display control unit 230 has been described as displaying the operating part OP, which is a part of the user U's body, on the display screen, but it is not limited to this. The display control unit 230 may also superimpose the image of the user U (for example, an image of the user U's whole body) captured by the imaging unit 52 in step S230 onto the image of the operating part OP and display it. With the sound and video output device 20 configured in this way, the user U is displayed on the display screen together with the instrument object P2, which can further enhance the user U's sense of ownership and familiarity with the instrument object P2.

[0140] [Interactive display of instrument object P2 based on user U's posture] Up to this point, we have described the case in which the display position of the instrument object P2 moves based on a predetermined movement trajectory OT2. That is, we have described how the display control unit 230 changes the position of the instrument object P2 on the display screen of the display device 51 in a manner that is not based on the posture of user U.

[0141] The instrument object P2 may move in response to the user U's playing operations. In this case, the display control unit 230 calculates the movement trajectory OT2 of the instrument object P2 each time (i.e., in real time) based on the user U's posture detected by the detection unit 210.

[0142] In other words, the display control unit 230 changes the position of the instrument object P2 on the display screen of the display device 51 in a manner that is based on the posture of the user U.

[0143] For example, the display control unit 230 causes the instrument object P2 to perform a predetermined movement when the user U assumes a specific posture (pose). For example, when the user U beckons or makes a heart shape with their fingers, the display control unit 230 moves the instrument object P2 closer to the display position of the user U's operating part OP.

[0144] In other words, when the user U is in a predetermined posture, the display control unit 230 changes the position of the instrument object P2 on the display screen of the display device 51 to bring it closer to the operation position PT.

[0145] For example, user U may be shorter than the display device 51. In such a case, if the instrument object P2 is above the display device 51 (in the direction where the display Y coordinate yd is smaller), user U's operating part OP may not be able to reach the display position of the instrument object P2. With the sound and video output device 20 configured as described above, when user U assumes a specific posture, the instrument object P2 moves closer to the operating part OP, so that user U can operate the instrument object P2 even if they are relatively short. With the sound and video output device 20 configured in this way, the game-like aspect of playing the instrument is improved, and user U can be given greater enjoyment.

[0146] [Collision detection of instrument object P2] (Step S250) The contact area determination unit 250 performs collision detection between the operating part OP and the instrument object P2 based on the position of the operating part OP on the display screen of the display device 51 and the display position of the instrument object P2.

[0147] In the example shown in the figure, the first operating area image P31 displayed at operating position PT1 is not in contact with the instrument object P2. The second operating area image P32 displayed at operating position PT2 is in contact with the instrument object P2. In this example, the contact area determination unit 250 determines that operating position PT1 is not in contact with the instrument object P2, and determines that operating position PT2 is in contact with the instrument object P2.

[0148] More specifically, the contact area determination unit 250 calculates the coordinates of the outer shape of the instrument object P2 in the display coordinate system based on the shape of the outer shape of the instrument object P2 indicated by the shape information D11 included in the instrument information D1 and the position of the instrument object P2 in the display coordinate system in which the instrument object P2 is displayed. The contact area determination unit 250 compares the calculated coordinates of the outer shape of the instrument object P2 with the coordinates of the operating part image P3. The contact area determination unit 250 determines that the instrument object P2 and the operating part image P3 have come into contact if the coordinates of the operating part image P3 coincide with any position of the coordinates of the outer shape of the instrument object P2.

[0149] As described above, the instrument object P2 is divided into multiple regions R. The instrument information D1 includes shape information D11 that shows the shape of the regions R. The contact region determination unit 250 refers to the shape information D11 and determines which of the multiple regions R of the instrument object P2 the operating part image P3 has come into contact with. Among the multiple regions R of the operating coordinate information D2, the region R that the operating part image P3 (i.e., the operating position PT of the user U) has come into contact with is also called the contact region RT.

[0150] In the example shown in the figure, the second operating area image P32 is in contact with region R2 of the instrument object P2. In this case, the contact region determination unit 250 determines that the operating area image P3 is in contact with region R2.

[0151] In other words, the contact area determination unit 250 determines, based on the operation coordinate information D2 and the shape information D11 of the instrument object P2, the contact area RT among the multiple areas R of the instrument object P2 displayed on the display screen of the display device 51, which is the area R that the operation position PT on the display screen of the display device 51, indicated by the operation coordinate information D2, has come into contact with.

[0152] As mentioned above, the operating area OP does not necessarily have to be a single specific location. For example, if the fingertips of the user U's right hand (first operating area OP1) and the fingertips of the left hand (second operating area OP2) are operating areas OP, the contact area determination unit 250 determines the contact area RT corresponding to the first operating area OP1 and the contact area RT corresponding to the second operating area OP2, respectively.

[0153] [Sound Output] (Step S260) Returning to Figure 14, the sound output unit 260 outputs a sound of the frequency assigned to the contact area RT determined in step S250 to the sound output unit 53. As a result, the sound output unit 53 outputs a sound of the frequency assigned to the contact area RT to the user U.

[0154] Furthermore, if there are multiple operating parts OP, such as the fingertips of the user U's right hand (first operating part OP1) and the fingertips of the left hand (second operating part OP2), the sound output unit 260 outputs a sound with a frequency assigned to the contact area RT corresponding to the first operating part OP1, and a sound with a frequency assigned to the contact area RT corresponding to the second operating part OP2, respectively.

[0155] As described above, each region R of the instrument object P2 is assigned a frequency corresponding to a musical scale. Therefore, depending on which of the multiple regions R (for example, regions R1 to R8) the manipulated area image P3 touches, a sound with a frequency corresponding to the region R is output. In other words, the instrument object P2 functions as an instrument that plays a musical scale according to the region R (contact region RT) that the manipulated area image P3 touches.

[0156] Furthermore, the sound output unit 260 outputs a sound with a tone corresponding to the instrument type information D12 included in the instrument information D1 acquired in step S210 to the sound output unit 53.

[0157] In other words, the sound output unit 260 outputs a sound that has a frequency assigned to the determined contact area RT and a timbre corresponding to the type of instrument indicated by the instrument information D1.

[0158] As described above, the type of instrument is selected based on the shape of the drawing P1 created by user U. In other words, the sound and video output device 20 can provide user U with the enjoyment and satisfaction of being able to play an instrument that they have created themselves on a large screen such as the display device 51.

[0159] [Control of Output Sound] The sound output unit 260 may change the characteristics of the sound based on the user U's posture. For example, the sound output unit 260 may change the frequency (i.e., pitch) based on the user U's right hand playing operation, and change the reverberation of the sound and the envelope curve showing the time change of volume based on the user U's left hand playing operation. In this example, the contact area RT is determined by the operation coordinate information D2 corresponding to the first operation part OP1 (e.g., the user U's right hand), and the output frequency is changed. In other words, in this example, the output frequency is not affected by the second operation part OP2 (e.g., the user U's left hand).

[0160] In other words, the sound output unit 260 changes the output frequency by changing the coordinate of the first operating part OP1, and changes the characteristics of the output sound by changing the coordinate of the second operating part OP2.

[0161] With the sound and video output device 20 configured in this way, it is possible to enrich the performance expression using the instrument object P2.

[0162] Furthermore, the sound output unit 260 may change the type of sound it outputs depending on the type of performance operation by the user U. For example, the sound output unit 260 may change the type of sound it outputs depending on the speed of change of the user U's operating part OP, taking a predetermined pose, the difference in color of an object the user U is wearing (for example, holding in their hand), etc.

[0163] In other words, the sound output unit 260 changes the mode of the sound it outputs according to at least one of the following: the rate of change of the coordinates of the operating part OP, the shape of the operating part OP, and the color of the operating part OP.

[0164] Furthermore, changes in the characteristics of sound include at least one change in the duration of the sound, the reverberation time of the sound, the shape of the sound's envelope curve, the composition of the sound's harmonic components, the composition of the chords, the degree of distortion of the sound's waveform, and the degree of modulation of the sound.

[0165] (Step S270) The calculation unit 200 determines whether it is necessary to end the performance of the instrument object P2. If the calculation unit 200 determines that the performance of the instrument object P2 should continue (Step S270; NO), it returns to step S220 and continues the processes described above. If the calculation unit 200 determines that the performance of the instrument object P2 should end (Step S270; YES), it terminates the series of sound and video output processes.

[0166] [Modified Version of Sound and Video Output Device 20: Performance by Multiple Users U] Figure 18 shows a modified version of the performance situation in this embodiment. Up to this point, the explanation has been given using the case where there is one user U as an example, but it is not limited to this. The sound and video output system 12a of this modified version makes it possible for multiple users U to perform the performance simultaneously. The figure shows a case where two users U, user U1 and user U2, are performing the performance simultaneously.

[0167] When multiple users U perform performance operations simultaneously, there are two possibilities: (1) multiple users U perform performance operations on a single instrument object P2 simultaneously, and (2) each user U displays a separate instrument object P2 (i.e., multiple instrument objects P2 are displayed simultaneously).

[0168] (1) When multiple users U simultaneously perform musical instrument operations on a single instrument object P2, in the sound and video output system 12a, in step S230 described above, the detection unit 210 detects the postures of each of the multiple users U without contact. In the example shown in the figure, the first operation part OP1 and the second operation part OP2, which are the operation parts OP of user U1, and the third operation part OP3 and the fourth operation part OP4, which are the operation parts OP of user U2, are detected, respectively.

[0169] In step S230 described above, the coordinate calculation unit 240 calculates the coordinates of the operation part image P3 displayed on the display screen of the display device 51 on the display screen of the display device 51 as operation coordinate information D2 for each user U, based on the detected posture.

[0170] In step S240 described above, the display control unit 230 superimposes and displays on the display screen of the display device 51 an image of an instrument object P2, which is an image based on the acquired instrument information D1 and is an image of a pattern corresponding to the shape of the drawing figure P1 (figure), and an image based on the detected posture and is an image showing the operating part OP which is at least a part of the user U's body, for each user U. In the example shown in the figure, the display control unit 230 displays the first operating part image P31 and the second operating part image P32 corresponding to the first operating part OP1 and the second operating part OP2 of user U1, and the third operating part image P33 and the fourth operating part image P34 corresponding to the third operating part OP3 and the fourth operating part OP4 of user U2.

[0171] In step S250 described above, the contact area determination unit 250 determines, for each user U, which of the multiple areas R of the instrument object P2 displayed on the display screen of the display device 51 is the area R that the operation position PT on the display screen of the display device 51, indicated by the operation coordinate information D2, has come into contact with, based on the operation coordinate information D2 and the shape information D11 of the instrument object P2.

[0172] In other words, the sound output unit 260 outputs a sound that has a frequency assigned to each contact area RT determined for each user U, and whose timbre corresponds to the type of instrument indicated by the instrument information D1. In this example, if there are multiple contact areas RT, the sound output unit 260 simultaneously outputs sounds of multiple frequencies corresponding to the contact areas RT. When one user U plays, only a single note can be output, but with the sound and video output system 12a configured in this way, when multiple users U play simultaneously, it becomes possible to output chords. In other words, with the sound and video output system 12a configured in this way, the manner of sound output when one user U plays can be made different from the manner of sound output when multiple users U play simultaneously. Therefore, the sound and video output system 12a can give users U the enjoyment of playing together in an ensemble.

[0173] In other words, the contact area determination unit 250 determines multiple contact areas RT where multiple operating positions PT indicated by the operating coordinate information D2 of multiple users U come into contact with a single instrument object P2 displayed on the display screen of the display device 51. The sound output unit 260 outputs a sound corresponding to each of the multiple contact areas RT determined for each user U for the single instrument object P2.

[0174] (2) When assigning each instrument object P2 to each of multiple users U, in step S210 described above, the instrument information acquisition unit 220 acquires the instrument information D1 generated by each of the multiple users U using the instrument information generation device 10, for each user U.

[0175] In other words, the instrument information acquisition unit 220 acquires instrument information D1, which is information generated based on the shape of the drawing figure P1 (figure) drawn by the user U's drawing operation, and which is associated with shape information D11 indicating the shape of the drawing figure P1 (figure), instrument type information D12 indicating the type of instrument based on the shape of the drawing figure P1 (figure), and frequency information D13 indicating the frequency assigned to each region of the drawing figure P1 (figure).

[0176] Furthermore, in step S220 described above, the display control unit 230 displays an instrument object P2 for each user U based on the instrument information D1 generated by each of the multiple users U.

[0177] In other words, the display control unit 230 superimposes an image based on the acquired instrument information D1, which is an instrument object P2 that is an image of a pattern corresponding to the shape of the drawing figure P1 (figure), and an operation part image P3 for each user U that is an image based on the detected posture and shows the operation part OP which is at least a part of the user U's body, and displays them on the display screen of the display device 51.

[0178] As described above, the type of instrument included in the instrument information D1 is selected based on the curvature of the lines constituting the drawn figure P1 (figure) and the complexity of the drawn figure P1 (figure), which is calculated based on the ratio of the distance from the start point to the end point of the lines constituting the drawn figure P1 (figure) to the length of the lines. The display control unit 230 displays instrument objects P2, which are different for each user U, in association with operation part images P3 for each user U. The contact area determination unit 250 determines the contact area RT for each user U based on the correspondence between the instrument object P2 and the operation part image P3 for each user U.

[0179] With the sound and video output system 12a configured in this way, multiple users U can perform performance operations simultaneously, thereby providing users U with greater enjoyment and satisfaction in the performance operation.

[0180] Furthermore, the sound and video output device 20 may output a different type of sound than when a single user U plays, if multiple users U play simultaneously.

[0181] For example, the sound output unit 260 may change at least one of the frequency and timbre depending on the number of detected users U.

[0182] Furthermore, the sound output unit 260 can output chords or control chord progressions in response to the user U's performance operations. For example, when a single user U outputs chords, different chords may be output depending on the difference in the collision detection area between the operation part OP and the instrument object P2. Also, for example, the sound output unit 260 may output chords of standard chord progressions that are considered pleasing in music theory (for example, repetition of tonic and dominant chords, or cadence resolution progressions such as dominant-seventh chord → tonic chord). Here, when multiple users U are performing performance operations simultaneously, the sound output unit 260 may produce richer chord structures and chord progressions compared to when a single user U is performing performance operations. For example, when multiple users U output chords, chords with a richer sound (for example, chords with added tension notes such as 7th or 9th) may be output compared to the chords output by a single user U.

[0183] In other words, the sound output unit 260 changes at least one of the following depending on the number of detected users U: the composition of the chords output simultaneously, or the progression of the chords output sequentially over time.

[0184] With the sound and video output system 12a configured in this way, when multiple users U perform the performance simultaneously, a richer musical expression is possible compared to when a single user U performs. Therefore, the sound and video output system 12a can provide users U with greater enjoyment and satisfaction compared to when a single user U performs.

[0185] Furthermore, when multiple users U perform performance operations simultaneously, the display mode of the operation part image P3 (also called a pointer image) displayed at the operation position PT may be made different for each user U. For example, the display control unit 230 may make the color and shape of the operation part image P3 different for each user U.

[0186] In other words, the display control unit 230 displays the pointer image on the display screen of the display device 51 in a manner that differs from that of the detected multiple users U, indicating the operation area OP on the display screen of the display device 51.

[0187] With the sound and video output system 12a configured in this way, when multiple users U perform performance operations simultaneously, it is possible to visually indicate which user U is operating the image P3 of the control part, making it easier to perform performance operations.

[0188] While embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and design changes and the like are also included within the scope of the gist of the present invention. For example, a computer program for realizing the functions of each of the above-described devices may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read by a computer system and executed. The term "computer system" as used herein may include hardware such as an operating system and peripheral devices.

[0189] Furthermore, "computer-readable recording media" refers to writable non-volatile memory such as flexible disks, magneto-optical disks, ROMs, and flash memory, portable media such as DVDs (Digital Versatile Discs), and storage devices such as hard disks built into computer systems. In addition, "computer-readable recording media" also includes volatile memory (such as DRAM (Dynamic Random Access Memory)) within computer systems that act as servers or clients when programs are transmitted via networks such as the Internet or communication lines such as telephone lines, which retains programs for a certain period of time.

[0190] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Also, the above program may be for the purpose of realizing a part of the functions described above. Furthermore, it may be a so-called differential file (differential program) that can realize the above functions in combination with a program already recorded in the computer system.

[0191] According to the present invention, it is possible to provide users with a performance experience that offers greater enjoyment and satisfaction.

[0192] 1...Performance system, 11...Instrument information generation system, 12, 12a...Audio and video output system

Claims

1. A detection unit that non-contactually detects the user's posture; an instrument information acquisition unit that acquires instrument information, which is generated based on the shape of a figure drawn by the user's drawing operation, and which associates shape information indicating the shape of the figure, instrument type information indicating the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each of the regions into which the figure is divided; a display control unit that superimposes and displays on a display screen an instrument object, which is an image based on the acquired instrument information, and an operation part image, which is an image based on the detected posture, indicating an operation part that is at least a part of the user's body; a coordinate calculation unit that calculates the coordinates on the display screen of the operation part image displayed on the display screen as operation coordinate information based on the detected posture; a contact area determination unit that determines, based on the operation coordinate information and the shape information of the instrument object, the contact area among the multiple regions of the instrument object displayed on the display screen that the operation position on the display screen indicated by the operation coordinate information has contacted; A sound-video output device comprising: a sound output unit that outputs a sound at a frequency assigned to the determined contact area, and having a timbre corresponding to the type of instrument indicated by the instrument information; 2. The sound-video output device according to claim 1, wherein the display control unit changes the position of the musical instrument object on the display screen.

3. The sound-video output device according to claim 1, wherein the operating area includes a first operating area and a second operating area, each corresponding to a plurality of parts of the user's body, and the sound output unit changes the frequency output by changing the coordinate of the first operating area and changes the manner of sound output by changing the coordinate of the second operating area.

4. The sound output device according to claim 1, wherein the sound output unit changes the manner of the sound it outputs according to at least one of the rate of change of the coordinates of the operating part, the shape of the operating part, and the color of the operating part.

5. The sound-video output device according to claim 3 or claim 4, wherein the change in the sound characteristics includes at least one change in the duration of the sound, the reverberation time of the sound, the shape of the sound envelope curve, the composition of the harmonic components of the sound, the composition of the chords, the degree of distortion of the sound waveform, and the degree of modulation of the sound.

6. A computer includes: a detection step for non-contact detection of the user's posture; an instrument information acquisition step for acquiring instrument information, which is generated based on the shape of a figure drawn by the user's drawing operation, and which associates shape information indicating the shape of the figure, the type of instrument based on the shape of the figure, and frequency information indicating the frequency assigned to each of the regions into which the figure is divided; a display control step for displaying on a display screen an image based on the acquired instrument information, which is an image of an instrument object with a pattern corresponding to the shape of the figure, and an operation part image based on the detected posture, which is an image showing an operation part that is at least a part of the user's body; a coordinate calculation step for calculating the coordinates on the display screen of the operation part image displayed on the display screen as operation coordinate information based on the detected posture; and a contact area determination step for determining a contact area, which is one of the multiple regions of the instrument object displayed on the display screen that the operation position on the display screen indicated by the operation coordinate information has come into contact with, based on the operation coordinate information and the shape information of the instrument object. A program for executing a sound output step which outputs a sound at a frequency assigned to the determined contact area, and whose timbre corresponds to the type of instrument indicated by the instrument information.

7. The sound-video output device according to claim 1, wherein the detection unit detects the posture of multiple users non-contactually, the contact area determination unit determines the contact area for each user, and the sound output unit outputs a sound for each contact area determined for each user that has a frequency assigned to the contact area and a tone corresponding to the type of instrument indicated by the instrument information.

8. The sound-video output device according to claim 7, wherein the contact area determination unit determines a plurality of contact areas where a plurality of operation positions indicated by the operation coordinate information of each of the plurality of users come into contact with a single instrument object displayed on the display screen, and the sound output unit outputs a sound corresponding to the plurality of contact areas determined for each user for a single instrument object.

9. The sound and video output device according to claim 7, wherein the type of instrument included in the instrument information is selected based on the curvature of the lines constituting the drawn figure and the complexity of the figure calculated based on the ratio of the distance from the start point to the end point of the lines constituting the figure to the length of the lines, the display control unit displays the instrument objects, each of which is different for each user, in association with the operation part image for each user, and the contact area determination unit determines the contact area for each user based on the correspondence between the instrument object and the operation part image for each user.

10. The sound output unit changes at least one of the frequency assigned to the contact area and the timbre corresponding to the type of instrument indicated by the instrument information, according to the number of detected users, as described in claim 7 or 8.

11. The sound-video output device according to claim 6, wherein the detection step is a step of detecting the posture of multiple users in a non-contact manner, the contact area determination step is a step of determining the contact area for each user, and the sound output step is a step of outputting a sound for each contact area determined for each user that has a frequency assigned to the contact area and a tone corresponding to the type of instrument indicated by the instrument information.

Images