Programs, methods, and information processing devices.

Abstract

To allow users to visually enjoy the performance. [Solution] The program displays an object in a virtual space based on performance information, and causes the computer to perform a process of moving the object upward from its displayed position as time progresses.

Description

Technical Field

[0001] The present disclosure relates to a program, a method, and an information processing apparatus.

Background Art

[0002] For example, there is known a computer graphics moving image generation system that can move an image object such as a dancer in synchronization with the performance of music such as a MIDI (Musical Instrument Digital Interface) piece, and generate a moving image that changes integrally with the progress of the music as well as the musical idea of the piece (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the computer graphics moving image generation system described in Patent Document 1, since it simply dynamically displays a character representing a dancer during the performance, improvement is desired in terms of visually entertaining the user with the performance.

[0005] In view of the above circumstances, an embodiment of the present disclosure aims to provide a program, a method, and an information processing apparatus that can visually entertain the user with the performance.

Means for Solving the Problems

[0006] A program according to an embodiment of the present disclosure causes a computer to execute a process of displaying an object in a virtual space based on performance information and moving the object upward from the displayed position over time.

Effects of the Invention

[0007] According to one embodiment of the present disclosure, a program, method, and information processing device are provided that can allow a user to visually enjoy a performance. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing a system according to one embodiment of the present disclosure. [Figure 2] This is a block diagram showing the configuration of an information processing device according to one embodiment of the present disclosure. [Figure 3] This figure shows an example of an image displayed on the screen of an information processing device according to one embodiment of the present disclosure. [Figure 4] This is an explanatory diagram regarding the user's perspective in a virtual space. [Figure 5] This diagram shows the appearance locations of objects in a virtual space. [Figure 6] This flowchart shows a process performed by an information processing device in one embodiment of the present disclosure. [Figure 7] This is a schematic diagram showing a system according to another embodiment of the present disclosure. [Figure 8] This is a schematic diagram showing a system according to another embodiment of the present disclosure. [Modes for carrying out the invention]

[0009] The following description relates to a program, method, and information processing apparatus according to one embodiment of the present disclosure. Common or corresponding elements are denoted by the same or similar reference numerals, and redundant descriptions are omitted or simplified as appropriate. In each figure, the configuration may be enlarged, reduced, or omitted as appropriate for the sake of explanation. In order to improve the visibility of the drawings, elements in the figures may be shown with lines other than solid lines (such as dashed lines or dotted lines) as necessary.

[0010] As shown in Figure 1, System 1 includes an information processing device 10 and an electronic musical instrument 20. The information processing device 10 is an example of a computer. The information processing device 10 can be, for example, a smartphone, a tablet device, or a PC (Personal Computer). In the example in Figure 1, a tablet device, which is an example of an information processing device 10, is placed on the music stand of the electronic musical instrument 20.

[0011] The electronic instrument 20 is, for example, an electronic keyboard. The electronic instrument 20 may also be other forms of electronic keyboard instruments such as electronic pianos, synthesizers, and electronic organs. The electronic instrument 20 may also be other forms of electronic instruments such as electronic percussion instruments, electronic wind instruments, and electronic string instruments.

[0012] As shown in Figure 2, the information processing device 10 includes a processor 100, a flash ROM (Read Only Memory) 110, a UI (User Interface) 120, a communication interface 130, and a display unit 140.

[0013] The processor 100 is, for example, a single processor or a multi-processor, and includes at least one processor. In a configuration including multiple processors, the processor 100 may be packaged as a single device, or it may consist of multiple physically separated devices within the information processing device 10. The processor 100 may be called, for example, a control unit, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), or an MCU (Micro Controller Unit).

[0014] The processor 100 includes a DSP (Digital Signal Processor), RAM (Random Access Memory), and the like. The processor 100 reads various programs and data stored in the flash ROM 110 and controls the information processing device 10 by using the RAM as a work area.

[0015] The flash ROM 110 is a non-volatile semiconductor memory such as a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically Erasable Programmable ROM). By the processor 100 executing an application 110A which is an example of a program, various processes (for example, various image processes) according to an embodiment of the present disclosure are executed.

[0016] The UI 120 is, for example, a mechanical switch, a touch panel, a mouse, a keyboard, etc. provided in the information processing apparatus 10. The UI 120 may include user interfaces such as gesture recognition and voice recognition. The UI 120 may include various HMIs (Human Machine Interfaces).

[0017] The communication interface 130 is, for example, an interface that communicably connects the information processing apparatus 10 to various external devices. External devices communicable with the information processing apparatus 10 are, for example, electronic musical instruments 20, smartphones, tablet terminals, PCs, web servers, clouds, etc.

[0018] A part of various processes according to an embodiment of the present disclosure may be executed by an external device connected to the information processing apparatus 10 via the communication interface 130, for example, instead of the processor 100.

[0019] A library 110B is stored in the flash ROM 110. In the library 110B, for example, objects (for example, CG (Computer Graphic) data) arranged in a three-dimensional virtual space VS (refer to FIG. 4 etc.) are registered. Various formats renderable as CG may be adopted for the format of the object. The processor 100 calls an object (for example, an object such as a tree or a hill, a background such as sky) from the library 110B and arranges it in the virtual space VS.

[0020] The display unit 140 includes a display composed of, for example, an LCD (Liquid Crystal Display) panel, an organic EL (Electro Luminescence) panel, or an LED (Light Emitting Diode) panel. The display may also be a touch panel display.

[0021] Figure 3 shows an example of an image displayed on the display unit 140 in response to the user U's performance operation on the electronic instrument 20. Figure 4 conceptually illustrates the relationship between the virtual space VS and the image displayed on the display unit 140. Figure 5 shows the position of the object 40 that appears in the virtual space VS in response to the performance operation. For example, the virtual space VS depicts a landscape with a undulating grassland with trees under the sky.

[0022] In the following explanation, the two mutually orthogonal horizontal directions in the virtual space VS world coordinate system are referred to as the X and Y directions. The vertical direction that is orthogonal to both the X and Y directions is referred to as the Z direction. That is, the X, Y, and Z directions are mutually orthogonal. For convenience, the axes extending in the X, Y, and Z directions may be referred to as the X-axis, Y-axis, and Z-axis, respectively. The X direction may also be called the front-back direction. The Y direction may also be called the left-right direction. The Z direction may also be called the up-down direction. Note that the directional names are used for convenience to explain the relative positional relationships of the constituent elements and do not indicate absolute directions. For example, the X direction may be defined as the left-right direction, and the Y direction may be defined as the front-back direction.

[0023] In Figure 4, the symbol P1 indicates the origin of the camera coordinate system. The origin P1 of the camera coordinate system initially coincides with the origin of the global coordinate system of the virtual space VS, for example, when application 110A is started. Similarly, the orientation D1 of the camera coordinate system initially coincides with the orientation of the global coordinate system. The origin P1 and orientation D1 of the camera coordinate system represent the viewpoint of user U in the virtual space VS. More specifically, the origin P1 of the camera coordinate system represents the viewpoint position of user U in the virtual space VS. The orientation D1 of the camera coordinate system represents the viewpoint direction of user U in the virtual space VS. For convenience, below, the origin P1 will be referred to as "viewpoint position P1," and the orientation D1 will be referred to as "viewpoint direction D1." "Viewpoint direction" may also be rephrased as "line of sight direction."

[0024] User U can change their viewpoint (i.e., viewpoint position P1 and viewpoint direction D1) in the virtual space VS by, for example, touching the display unit 140, which is a touch panel display. The processor 100 updates the origin P1 and orientation D1 of the camera coordinate system in response to such touch operations.

[0025] The processor 100 converts the global coordinates to the camera coordinate system. The processor 100 then performs a projection transformation on the converted camera coordinate system. By performing a viewport transformation, the processor 100 displays the two-dimensional image projected onto the projection surface within a viewport, which is a designated area on a window represented in the screen coordinate system. The processor 100 performs these coordinate transformations sequentially. In Figure 4, the symbol S1 conceptually represents the screen area of ​​the display unit 140 defined in the screen coordinate system.

[0026] As the user U's viewpoint in the virtual space VS (i.e., viewpoint position P1 and viewpoint direction D1) is updated in response to touch operations, the area within the virtual space VS displayed on the screen area S1 is also updated. For example, when user U slides their finger that touched the display unit 140 to the left, the camera coordinate system rotates clockwise around the Z axis, which is the yaw axis. As the viewpoint direction D1 rotates clockwise around viewpoint position P1 (in other words, the viewpoint direction D1 pans to the right), user U can observe the area within the virtual space VS to the right of their own perspective.

[0027] The electronic instrument 20 outputs performance information to the information processing device 10 in response to performance operations performed by the user U on the electronic instrument 20. This performance information is, for example, a MIDI signal. Performance operations include, for example, pressing keys on the keyboard of the electronic instrument 20. Pressing keys is one example of an operation that generates musical tones.

[0028] Performance information is input to the processor 100 via the communication interface 130. The processor 100 places objects 40 in the virtual space VS based on the input performance information. For example, each time performance information is input (for example, each time user U performs a key press), the processor 100 places a new object 40 in the virtual space VS. More specifically, each time performance information is input, the processor 100 places a predetermined number (one or more predetermined numbers) or a random number determined by a random function or the like into the virtual space VS. The processor 100 may also place a number of objects 40 in the virtual space VS according to the performance content (for example, pitch, velocity, number of chords, etc.). As an example, the processor 100 places more objects 40 in the virtual space VS the higher the velocity value of the key press. The processor 100 may place objects 40 in the virtual space VS at a frequency of once every few times, rather than every time performance information is input.

[0029] The processor 100 places object 40 at a random location. More specifically, the processor 100 determines the position in the virtual space VS based on a random number generated using a random function and the pitch of a musical note specified by a key press (for example, a note number included in a MIDI signal). The processor 100 places object 40 at the determined location.

[0030] The processor 100 randomly places objects 40 within a sector-shaped region R1 of ±75 degrees (totaling 150 degrees) relative to the viewpoint direction D1, as shown in the illustrative schematic diagram Figure 5. The sector-shaped region R1 corresponds, for example, to a person's field of view (including the effective field of view and peripheral field of view). The angular values ​​indicating the size of the sector-shaped region R1 are merely examples; they may be other values.

[0031] Specifically, the processor 100 determines an azimuth angle (first direction D11) between 0 and 150 degrees, where the viewpoint direction D1 is +75 degrees, based on, for example, the pitch of a specified musical note. If the electronic instrument 20 has 88 keys, the processor 100 determines a different first direction D11 for each pitch in increments of 150 / 87 degrees. For example, when the third lowest pitch key is pressed, the first direction D11 is determined to be in the direction of 2 × (150 / 87) degrees. For example, when the fourth highest pitch key is pressed, the first direction D11 is determined to be in the direction of 84 × (150 / 87) degrees.

[0032] Any reference to elements using designations such as “First,” “Second,” etc., as used in this disclosure, does not generally limit the quantity or order of those elements. These designations are used for convenience to distinguish between two or more elements. Therefore, references to the First and Second elements do not imply, for example, that only two elements are adopted, or that the First element must precede the Second element.

[0033] The processor 100 sets a rectangular region R11 with edges extending in the determined first direction D11 and in the perpendicular direction (second direction D12) that is orthogonal to the first direction D11. The length of each edge of the rectangular region R11 is set in advance, for example, assuming the field of view and visual acuity of a typical person. The processor 100 randomly places objects 40 within the rectangular region R11.

[0034] In this way, the processor 100 places the object 40 in the virtual space VS according to the position of the pressed key on the keyboard. Furthermore, the processor 100 randomly places the object 40 within a rectangular region R11 determined based on the pitch of the musical note. Thus, the position of the object 40 is determined to some extent according to the pitch of the musical note. More specifically, the horizontal position of the object 40 is determined in a direction corresponding to the pitch of the musical note, while the distance from the viewpoint position P1 to the object 40 and the vertical position of the object 40 are determined randomly. Because the pitch of the performance is reflected in the appearance position of the object 40 (i.e., not completely random and not too mechanical), the user U can visually enjoy the performance expression and find it pleasant.

[0035] The processor 100 may randomly place objects 40 in all directions centered on the user U's viewpoint position P1. In other words, user U may randomly place objects 40 not only within the field of view but also outside the field of view.

[0036] Object 40 represents, for example, a triangular prism-shaped light. Object 40 is not limited to a triangular prism shape; it may also be cylindrical or rectangular. Thus, object 40 is a prism or cylinder in which the vertical length is longer than the horizontal length. Object 40 is not limited to a prism shape; it may also be a sphere or other shape. Object 40 is not limited to light; it may also represent other things such as stars. In this embodiment, by using a simple 3D model such as a triangular prism as object 40, there is an advantage in that the processing load on the processor 100 to render the image (which is necessary for 3D modeling) is reduced compared to when a complex 3D model is used. Therefore, changes in the image corresponding to the performance information are reflected with a short response (in other words, the rendering process is executed with low latency in response to the input of performance information).

[0037] Furthermore, all objects 40 placed in the virtual space VS are identical (for example, all are the same in size and shape). The objects 40 are randomly placed near and far from the user U's viewpoint position P1. Therefore, as shown in Figure 3, for example, the objects 40 that appear in the image of the virtual space VS differ not only in position but also in size due to perspective. Objects 40 of the same shape appear in various places and at various sizes within the image. Therefore, for example, even though the image displays a large number of objects 40 in a disordered manner, it gives a sense of unity. In addition, in this embodiment, by randomly placing the same objects 40 near and far from the user U's viewpoint position P1, and also randomly in the height direction, the shape and position of the objects 40 that the user U sees are varied, keeping the user from getting bored while watching the image. Since all objects 40 placed in the virtual space VS are identical and do not require complex calculation processing, there is the advantage that the processing load on the processor 100 to render the image is reduced. Therefore, changes in the video corresponding to the performance information are reflected with a short response time (in other words, rendering processing is performed with low latency in response to the input of performance information).

[0038] Object 40 may change depending on the performance content (e.g., pitch, velocity, number of chords, etc.). For example, the higher the velocity value of the key press, the larger the size of Object 40. For example, the higher the pitch, the brighter Object 40 becomes. For example, the higher the pitch, the cooler the color of Object 40 will be displayed, and the lower the pitch, the warmer the color of Object 40 will be displayed.

[0039] The objects 40 do not have to be the same size or shape. For example, multiple types of objects 40 with different sizes and shapes may be placed in the virtual space VS. The types of objects 40 to be placed may be determined randomly. The types of objects 40 to be placed may also be determined according to the performance content (e.g., pitch, velocity, number of chords, etc.).

[0040] The processor 100 changes the upward movement speed of object 40 after a set time has elapsed from the moment object 40 is displayed. In other words, when object 40 appears in the virtual space VS (or, in other words, when it is placed in the virtual space VS), it moves upward from its appearance position as time progresses. Image 30b in Figure 3 is an image taken a few seconds after image 30a in Figure 3. As can be seen by comparing images 30a and 30b, object 40 has moved upward.

[0041] Object 40 moves upward and disappears at a speed that, for example, represents the envelope of a musical note. Specifically, object 40 slowly rises from its appearance point for a few seconds (e.g., 2-3 seconds) (rising at the first velocity), then accelerates to a second velocity faster than the first velocity for the next few seconds (e.g., 2-3 seconds) before disappearing. The rise at the first velocity represents, for example, the attack to the sustain. The acceleration to the second velocity and the subsequent disappearance represent, for example, the release.

[0042] In this embodiment, all objects 40 move in the same way. In another embodiment, the movement of the objects 40 may change depending on the performance content (e.g., pitch, velocity, number of chords, etc.). For example, the duration of the increase at the first velocity may change depending on the performance content. For example, the time it takes to reach the second velocity from the first velocity may change depending on the performance content. As an example, the higher the velocity value of the key press, the shorter the time it takes to reach the second velocity from the first velocity (in other words, the acceleration increases).

[0043] Various forms of expression are possible through the movement of object 40. Effects such as object 40 exploding like fireworks upon its disappearance may be added. Object 40's movement may also express something other than the musical sound envelope.

[0044] According to this embodiment, user U can enjoy the visuals of the virtual space VS and have an experience that cannot be obtained by simply playing the electronic instrument 20. Because the visuals of the virtual space VS change in accordance with the performance (for example, objects 40 appear in response to key presses), user U can visually enjoy the musical expression. By watching the objects 40 appear, rise, and disappear one after another with each key press, user U can visually enjoy the performance and play with pleasure. The rising of the objects also lifts user U's spirits. Therefore, by watching the movement of the objects 40, which represent the envelope of the musical sound, user U can enjoy the performance even more visually.

[0045] User U can, for example, find the act of playing music enjoyable through the visuals of the virtual space VS, which can lead to increased motivation for practicing. Others can enjoy User U's performance not only aurally but also visually. Even people with hearing impairments can enjoy playing music through the visuals of the virtual space VS.

[0046] Using Figure 6, a process executed by the information processing device 10 (processor 100) in one embodiment of this disclosure will be described. For example, when application 110A is started, the execution of the process shown in Figure 6 begins. For example, when application 110A is terminated, the process shown in Figure 6 ends.

[0047] The steps in the flowchart shown in this embodiment may be rearranged, provided they are consistent with each other. For example, while this disclosure presents various steps in an exemplary order, it is not limited to this order. Furthermore, the steps in the flowchart shown in this embodiment may be executed in parallel or concurrently, provided they are consistent with each other.

[0048] As shown in Figure 6, the information processing device 10 performs various initial settings (step S101). For example, the information processing device 10 sets the origin of the 3D global coordinate system to the virtual space VS. The information processing device 10 calls each default object (such as a tree object) from the library 110B. The information processing device 10 performs a modeling transformation on each of the called objects, which are represented in the local coordinate system, and places them in the global coordinate system. That is, the information processing device 10 renders each default object in the virtual space VS. The information processing device 10 further places the origin of the camera coordinate system at a predetermined position in the virtual space VS.

[0049] The information processing device 10 performs a projection transformation on the camera coordinates using internal parameters. To display the two-dimensional image projected onto the projection surface within a viewport, which is a designated area on a window represented in the screen coordinate system, the information processing device 10 performs a viewport transformation. The information processing device 10 performs these coordinate transformations sequentially. As a result, the image of the virtual space VS is displayed on the display unit 140.

[0050] The information processing device 10 waits for input of performance information (step S102). The input performance information is, for example, information corresponding to the performance of the electronic instrument 20 by user U (such as MIDI signals). The input performance information is not limited to the performance by user U, but may also be pre-prepared music data.

[0051] When performance information is input from the electronic instrument 20 (step S102: YES), the information processing device 10 determines the position where object 40 will appear (step S103). Illustratively, as explained with reference to Figure 5, the information processing device 10 determines the appearance position of object 40 based on random numbers generated using a random function and the pitch of the musical note specified by the key press operation (for example, the note number included in the MIDI signal). That is, the information processing device 10 determines the display position of object 40 in the virtual space VS based on the pitch of the musical note included in the performance information, and also determines the display position of object 40 in the virtual space VS based on random numbers generated using a random function. The information processing device 10 places object 40 at the determined appearance position (step S104). As a result, object 40 is displayed. For example, as object 40 is placed in the virtual space VS each time a key is pressed, the image 30a shown in Figure 3 is displayed on the display unit 140.

[0052] The information processing device 10 moves the object 40, which was placed in the virtual space VS in step S104, upward (in other words, upward) (step S105). The information processing device 10 then removes the object 40 that has been moved upward from the virtual space VS (step S106). For example, the information processing device 10 moves the object 40 upward at a speed that represents the envelope of a musical tone and then removes it from the virtual space VS. The information processing device 10 repeats the processes in steps S102 to S106, for example, until application 110A is terminated.

[0053] The above is a description of exemplary embodiments of the present disclosure. Embodiments of the present disclosure are not limited to those described above, and various modifications are possible within the scope of the technical idea of ​​the present disclosure. For example, embodiments of the present application include combinations of embodiments explicitly shown in the specification or obvious embodiments as appropriate.

[0054] In the above embodiment, the video of the virtual space VS is displayed on the screen of a portable device such as a tablet terminal placed on a music stand. However, in another embodiment, as shown in Figure 7, the video of the virtual space VS may be displayed on a stationary large display connected to a PC or the like. In the example in Figure 7, the video of the virtual space VS is displayed on a triple display. Compared to the above embodiment, user U can experience an even greater sense of presence and immersion.

[0055] As shown in Figure 8, the information processing device 10 may be an XR (Cross Reality) device such as an HMD (Head Mounted Display). The information processing device 10 is, for example, an image device that allows user U to observe images of a virtual world or images that fuse the real world and the virtual world. User U can play the electronic musical instrument 20 while viewing images of the virtual space VS displayed on the XR device or images that fuse objects from the real world and the virtual world. To make it easier for user U to play the electronic musical instrument 20, the information processing device 10 displays the electronic musical instrument 20 in a pass-through manner, for example. Even when using an XR device, user U can experience an even greater sense of presence and immersion compared to the above embodiment.

[0056] The information processing device 10 is not limited to an HMD; it may also be other forms of wearable devices such as smart glasses. These types of wearable devices are sometimes called VR (Virtual Reality) goggles, AR (Augmented Reality) goggles, MR (Mixed Reality) goggles, glasses displays, or glasses devices. [Explanation of Symbols]

[0057] 1: System, 10: Information Processing Unit, 20: Electronic Musical Instrument, 100: Processor, 110A: Application, 110B: Library, 140: Display Unit

Claims

1. Based on performance information, objects are displayed in a virtual space. The object is moved upward from the displayed position as time progresses. To have the computer perform the process. program.

2. Based on the pitch of the musical notes included in the performance information, the display position of the object in the three-dimensional virtual space is determined. The program according to claim 1.

3. The aforementioned object is a prism or cylinder in which the vertical length is longer than the horizontal length. The program according to claim 1.

4. Each time a signal corresponding to a performance operation on an electronic musical instrument is input, a new object is displayed in the three-dimensional virtual space. The program according to claim 1.

5. After a set time has elapsed from the time the object is displayed, the upward movement speed of the object is changed. The program according to claim 1.

6. The display position of the object in the virtual space is determined based on random numbers generated using a random function. The program according to any one of claims 1 to 5.

7. Based on performance information, objects are displayed in a virtual space. The object is moved upward from the displayed position as time progresses. To have the computer perform the process. method.

8. Based on performance information, objects are displayed in a virtual space. The object is moved upward from the displayed position as time progresses. Equipped with a control unit, Information processing device.

Images