Methods, programs, and information processing devices

Abstract

The process of aligning objects equivalent to real-world electronic musical instruments with real-world electronic musical instruments, using precise and simple operations, and then placing them in a virtual space. [Solution] The method involves a computer detecting the coordinates of the user's fingers that performed the operation on an electronic musical instrument's control element when a signal corresponding to the user's operation on the control element is input, based on an image captured by an imaging device, and then, based on the detected coordinates, arranging objects of the electronic musical instrument in the virtual space on a display capable of superimposing the image captured by the imaging device and the virtual space, so as to correspond to the position of the electronic musical instrument in the real world within the image.

Description

[Technical Field]

[0001] This disclosure relates to methods, programs, and information processing devices. [Background technology]

[0002] A technology for placing musical instruments in a virtual space is described, for example, in Patent Document 1. In Patent Document 1, the user wears a device that can track hand movements, such as a glove-type controller, and opens their hand and holds it still for several seconds on a flat surface, such as a desk, in the real world. The monitor of the HMD (Head Mounted Display) displays piano keys in the virtual space according to the position where the hand was held still.

[0003] Technologies that create a UX (User Experience) that fuses the real and virtual worlds include, for example, MR (Mixed Reality) and AR (Augmented Reality). Using these types of technologies, for example, when a user wearing an XR (Cross Reality) device such as an HMD plays an electronic musical instrument in the real world, visual effects corresponding to the playing operation can be generated around the electronic instrument and shown to the user. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Patent No. 6419932 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] To provide such a user experience, for example, there is a need for technology that can align objects equivalent to real-world electronic musical instruments with real-world electronic instruments and place them in a virtual space with more precise and simpler operation.

[0006] In view of the above circumstances, embodiments of this disclosure aim to provide a method, program, and information processing device that can accurately and easily align objects corresponding to real-world electronic musical instruments with real-world electronic musical instruments and place them in a virtual space. [Means for solving the problem]

[0007] In one embodiment of the present disclosure, a computer, upon receiving a signal corresponding to a user's operation on an electronic musical instrument's control element, detects the coordinates of the user's finger that performed the operation on the control element based on an image captured by an imaging device, and, based on the detected coordinates, places objects of the electronic musical instrument in a virtual space on a display capable of superimposing the image captured by the imaging device and the virtual space, so as to correspond to the position of the electronic musical instrument in the real world within the image. [Effects of the Invention]

[0008] According to one embodiment of the present disclosure, a method, program, and information processing device are provided that can align an object corresponding to a real-world electronic musical instrument with a real-world electronic musical instrument using accurate and simple operation and place it in a virtual space. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing a system according to one embodiment of the present disclosure. [Figure 2] This block diagram shows the configuration of an XR device according to one embodiment of the present disclosure. [Figure 3] This diagram schematically illustrates one example of a method for aligning musical instrument objects with real-world electronic instruments and placing them in a virtual space. [Figure 4] This figure shows an example of a virtual effect that is displayed and merged with the real world when a user wearing an XR device plays an electronic musical instrument, according to one embodiment of this disclosure. [Figure 5] This flowchart shows a process performed by an XR device in one embodiment of this disclosure. [Figure 6] This diagram schematically illustrates one example of a method for aligning musical instrument objects with real-world electronic instruments and placing them in a virtual space. [Figure 7] This diagram schematically illustrates one example of a method for aligning musical instrument objects with real-world electronic instruments and placing them in a virtual space. [Figure 8] This diagram schematically illustrates one example of a method for aligning musical instrument objects with real-world electronic instruments and placing them in a virtual space. [Figure 9] This diagram schematically illustrates one example of a method for aligning musical instrument objects with real-world electronic instruments and placing them in a virtual space. [Modes for carrying out the invention]

[0010] The following description relates to a method, program, and information processing apparatus according to one embodiment of the present disclosure. Common or corresponding elements are denoted by the same or similar reference numerals, and redundant descriptions are omitted or simplified as appropriate. In each figure, the configuration may be enlarged, reduced, or omitted as appropriate for the sake of explanation. In order to improve the visibility of the drawings, elements in the figures may be shown with lines other than solid lines (such as dashed lines or dotted lines) as necessary.

[0011] As shown in FIG. 1, the system 1 includes an XR device 10 and an electronic musical instrument 20. The XR device 10 is, for example, an imaging device (which may also be referred to as a “video device”) that can allow the user U to observe an image that combines the real world and the virtual world. It is also an example of an information processing device that is a computer. In the present embodiment, the XR device 10 is an HMD. The XR device 10 may also be other forms of wearable devices such as smart glasses. This type of wearable device may also be called a VR (Virtual Reality) goggle, an AR goggle, a glass display, a glass device, etc. The XR device 10 is not limited to wearable devices. The XR device 10 may also be a smartphone, a tablet terminal, or a PC (Personal Computer) having a photographing function on which an MR app or an AR app is installed.

[0012] The XR device 10 is, for example, of the video see-through type. The XR device 10 can photograph the real space with a camera and allow the user U to observe an image in which an object is synthesized in the photographed real space. The XR device 10 is not limited to the video see-through type and may also be of the optical see-through type.

[0013] The electronic musical instrument 20 is an electronic device that can output a MIDI (Musical Instrument Digital Interface) signal. In the present embodiment, the electronic musical instrument 20 is, for example, an electronic keyboard. The electronic musical instrument 20 may also be other forms of electronic keyboard musical instruments such as an electronic piano, a synthesizer, or an electronic organ. The electronic musical instrument 20 may also be other forms of electronic musical instruments such as an electronic percussion instrument, an electronic wind instrument, or an electronic string instrument.

[0014] As shown in FIG. 2, the XR device 10 includes a processor 100, a flash ROM (Read Only Memory) 110, a UI (User Interface) 120, a communication interface 130, a 6-axis sensor 140, a gaze detection sensor 150, a camera 160, and a display element 170.

[0015] The processor 100 is, for example, a single processor or a multi-processor, and includes at least one processor. In a configuration including multiple processors, the processor 100 may be packaged as a single device, or it may consist of multiple physically separated devices within the XR device 10. The processor 100 may be called, for example, a control unit, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), or an MCU (Micro Controller Unit).

[0016] The processor 100 includes a DSP (Digital Signal Processor), RAM (Random Access Memory), and other components. The processor 100 reads various programs and data stored in the flash ROM 110 and controls the XR device 10 by using the RAM as a work area.

[0017] The flash ROM 110 is a non-volatile semiconductor memory such as flash memory, EPROM (Erasable Programmable ROM), or EEPROM (Electrically Erasable Programmable ROM). The processor 100 executes the control program 110A, thereby performing various processes according to one embodiment of this disclosure (for example, various coordinate transformations, various tracking, object manipulation, etc.).

[0018] UI120 is, for example, a mechanical switch or hand controller provided on the XR device 10. UI120 is not limited to a hardware-based user interface; it may also be a software-based user interface such as gesture recognition or voice recognition. UI120 may also include various HMIs (Human Machine Interfaces).

[0019] The communication interface 130 is, for example, an interface that enables communication between the XR device 10 and various external devices. Exemplarily, the communication interface 130 enables mutual communication between the XR device 10 and external devices via wireless communication standards such as Bluetooth®, Wi-Fi, IR (infrared) communication, or via a wired cable. Examples of external devices that can communicate with the XR device 10 include MIDI devices such as electronic musical instruments 20, smartphones, tablet devices, PCs (Personal Computers), web servers, and cloud services.

[0020] Some of the various processes according to one embodiment of this disclosure may be performed by an external device connected to the XR device 10 via a communication interface 130, instead of the processor 100. For example, various coordinate transformations and object library management may be performed by the external device.

[0021] The 6-axis sensor 140 is an IMU (Inertial Measurement Unit). The 6-axis sensor 140 detects acceleration in the x, y, and z axes, as well as angular velocity in pitch, roll, and yaw. The processor 100 can, for example, detect the posture of the user U wearing the XR device 10 based on the output of the 6-axis sensor 140.

[0022] The gaze detection sensor 150 detects the gaze direction of the user U wearing the XR device 10. The processor 100 can perform eye tracking, for example, based on the output of the gaze detection sensor 150.

[0023] The camera 160 includes, for example, a pair of camera modules positioned to correspond to the left and right eyes. The camera module corresponding to the right eye captures the area around user U wearing the XR device 10 with a field of view seen from the right eye. The camera module corresponding to the left eye captures the area around user U wearing the XR device 10 with a field of view seen from the left eye. The processor 100 acquires disparity images using these camera modules.

[0024] The flash ROM 110 stores library 110B. Library 110B contains, for example, objects (e.g., CG (Computer Graphic) data) that mimic the electronic keyboards of various models. Various formats that can be rendered as CG can be used for the object format. Hereafter, electronic musical instruments in the virtual world will be referred to as "instrument object 30". Instrument object 30 is an example of a first object that represents a specific model of electronic musical instrument 20 in the real world.

[0025] The processor 100 calls an instrument object 30 from the library 110B. The user can specify the instrument object 30 to be called, for example, by operating on the UI 120. The processor 100 may also identify the model of the electronic instrument 20 placed in front of the user U using image recognition and call the instrument object 30 corresponding to the identified model from the library 110B.

[0026] The display element 170 includes, for example, a pair of display elements corresponding to the left and right eyes. The processor 100 synthesizes the instrument object 30 with the captured image corresponding to the right eye and outputs this synthesized image to the display element corresponding to the right eye. The processor 100 synthesizes the instrument object 30 with the captured image corresponding to the left eye and outputs this synthesized image to the display element corresponding to the left eye. The display element corresponding to the right eye displays the input synthesized image to the right eye of user U wearing the XR device 10. The display element corresponding to the left eye displays the input synthesized image to the left eye of user U wearing the XR device 10. As a result, user U can observe an image in which the instrument object 30 is superimposed on the real space around them. The display element 170 is an example of a display that can superimpose and display an captured image and a virtual space.

[0027] Figure 3 schematically shows an example of how to align and place the instrument object 30 in the virtual space relative to the electronic instrument 20. In Figure 3, the instrument object 30 is an object that mimics the electronic instrument 20, and has the same number of keys as the electronic instrument 20. For convenience, the instrument object 30 is shown with a dashed line.

[0028] For example, a virtual screen is superimposed in front of the eyes of user U, who is wearing the XR device 10. Guidance is displayed on this virtual screen. In the example in Figure 3, user U follows the guidance and presses the keys at both ends of the keyboard of the electronic instrument 20. Specifically, user U presses the highest key on the keyboard of the electronic instrument 20 with the index finger of their right hand and the lowest key with the index finger of their left hand. The timing of pressing the two keys located at both ends can be simultaneous or slightly delayed (for example, by a few seconds). For convenience, the highest key on the keyboard of the electronic instrument 20 is labeled "Key KH". The lowest key on the keyboard of the electronic instrument 20 is labeled "Key KL".

[0029] The XR device 10 and the electronic instrument 20 are connected in a way that allows them to communicate with each other. The electronic instrument 20 sends MIDI signals to the XR device 10 in response to key presses on the keys at both ends of the keyboard. For convenience, the MIDI signal corresponding to a key press on key KH is denoted as "MIDI signal SH". The MIDI signal corresponding to a key press on key KL is denoted as "MIDI signal SL".

[0030] The XR device 10 continuously performs hand tracking, for example. Exemplarily, the XR device 10 tracks the movement of various parts of the hand, including the fingertips of the index fingers, using images captured by the camera 160. The XR device 10 may also perform hand tracking using hand tracking sensor devices. That is, the XR device 10 may communicate with hand tracking sensor devices attached to the index fingers of each hand to track the movement of the fingertips.

[0031] The XR device 10 detects hand tracking information of the right index finger tip when it receives the MIDI signal SH. That is, the XR device 10 detects the coordinates (3D coordinates) of the right index finger tip at the time it receives the MIDI signal SH. The XR device 10 detects hand tracking information of the left index finger tip when it receives the MIDI signal SL. That is, the XR device 10 detects the coordinates (3D coordinates) of the left index finger tip at the time it receives the MIDI signal SL. For convenience, the detected coordinates of the right index finger tip are denoted as "coordinate CH". The detected coordinates of the left index finger tip are denoted as "coordinate CL".

[0032] For convenience, the coordinates of the highest key among the keys of instrument object 30 are "coordinate CH". V It is written as follows: The coordinates of the lowest key on the keyboard of instrument object 30 are "coordinates CL V It is written as follows:

[0033] The XR device 10 controls the coordinates CH. V Align it to coordinate CH and coordinate CL V The object is aligned to coordinate CL. In other words, the XR device 10 aligns the instrument object 30 to the real-world electronic instrument 20 based on the coordinates of the fingertips of each index finger detected by hand tracking.

[0034] The XR device 10 places the aligned instrument object 30 in the virtual space. That is, the XR device 10 composites the instrument object 30 onto the captured image corresponding to the right eye and the captured image corresponding to the left eye, respectively, so that the instrument object 30 is displayed at the aligned position. As a result, user U can observe an image in which the instrument object 30 is superimposed on the real-world electronic instrument 20.

[0035] User U can position and place the instrument object 30 in the virtual space with a simple operation, such as pressing a key (pressing a control that is originally attached to the electronic instrument 20). Furthermore, User U can position and place the instrument object 30 in the virtual space with the natural action of playing the electronic instrument 20.

[0036] In the example shown in Figure 3, the coordinates of the two furthest keys on the keyboard are used for alignment. Using the coordinates of two points that are far apart improves the accuracy of the alignment. The keys being operated are at both ends and are in a physically easy-to-understand position. Therefore, for example, even if user U is unfamiliar with keyboard operation, they can press them without hesitation.

[0037] Figure 4 shows an example of a virtual effect that is displayed and blended into the real world when user U, wearing the XR device 10, plays the electronic instrument 20. In the example in Figure 4, when a key is pressed on the electronic instrument 20, the electronic instrument 20 sends a MIDI signal to the XR device 10. The XR device 10 obtains the note number contained in the MIDI signal and identifies the key with the same note number from among the keys of the instrument object 30. The XR device 10 displays a visual effect EF above the identified key of the instrument object 30.

[0038] In the example shown in Figure 4, an effect EF is displayed where sparkling particles are generated from the key pressed by user U, rise, and disappear. User U can experience the UX of playing a real-world electronic instrument 20 while generating particles on the keyboard.

[0039] In the example shown in Figure 4, the electronic instrument 20 and the instrument object 30 are offset in the depth direction. This is because the position of the fingertip when the key is pressed is the assumed position (i.e., the pre-set coordinate CH). V , coordinate CL VThis is because there is a misalignment. Each key is long in the depth direction. Therefore, such a misalignment in the depth direction can occur. On the other hand, each key is short in the width direction (the left-right direction in which the individual keys are lined up). Therefore, in the left-right direction, there is less of a discrepancy between the position of the fingertip when the key is pressed and the expected position. Because the left-right misalignment between the electronic instrument 20 and the instrument object 30 is small, the effect EF can be generated with almost no deviation from the key pressed by user U.

[0040] Using Figure 5, a process performed by the XR device 10 (processor 100) in one embodiment of this disclosure will be described. For example, when the power of the XR device 10 is turned on, the execution of the process shown in Figure 5 begins.

[0041] The steps in the flowchart shown in this embodiment may be rearranged, provided they are consistent with each other. For example, while this disclosure presents various steps in an exemplary order, it is not limited to this order. Furthermore, the steps in the flowchart shown in this embodiment may be executed in parallel or concurrently, provided they are consistent with each other.

[0042] As shown in Figure 5, the XR device 10 performs various initial settings (step S101). For example, the XR device 10 sets its position at startup to the origin of the 3D global coordinate system (i.e., sets the origin of the camera coordinate system to the origin of the global coordinate system). The XR device 10 monitors the output of the 6-axis sensor 140, for example, and sequentially updates its attitude and position. The attitude (rotation matrix) and position (translation vector) of the XR device 10 are used as external parameters for converting global coordinates to 3D camera coordinates. These external parameters may include not only the information obtained from the 6-axis sensor 140, but also the viewpoint direction information obtained from eye tracking. "Viewpoint direction" can be rephrased as "gaze direction".

[0043] The XR device 10 performs projection transformations on the camera coordinates using internal parameters. To display the two-dimensional image projected onto the projection surface within a viewport, which is a specified area on a window represented in the screen coordinate system, the XR device 10 performs viewport transformations. The XR device 10 performs these coordinate transformations sequentially. As a result, user U can view the real-world image displayed on the XR device 10 with both eyes, and can also view the image that fuses the real world and virtual world objects (such as the musical instrument object 30) with both eyes.

[0044] The XR device 10 waits for an operation to transition to calibration mode (step S102). Calibration mode is a mode in which the instrument object 30 is aligned with the real-world electronic instrument 20. When a predetermined user operation is received by the UI 120 (step S102: YES), the XR device 10 transitions to calibration mode (step S103).

[0045] The XR device 10 waits for a MIDI signal input (step S104). In other words, the XR device 10 waits for a key press operation on the electronic instrument 20. In the example in Figure 3, the XR device 10 waits for MIDI signal SH and MIDI signal SL inputs.

[0046] When transitioning to calibration mode, the XR device 10 may, for example, display a guidance screen in front of the user U, fused with real-world images. The user U can then follow the guidance and perform the necessary operations to align the instrument object 30 with the electronic instrument 20.

[0047] When a MIDI signal (more specifically, MIDI signal SH and MIDI signal SL) is received (step S104: YES), the XR device 10 detects the corresponding 3D coordinates (step S105). The XR device 10 can determine from the note number contained in the MIDI signal whether the received signal is MIDI signal SH, and whether the received signal is MIDI signal SL.

[0048] In the example shown in Figure 3, the XR device 10 detects coordinates CH and CL. These detected coordinates CH and CL are, for example, screen coordinates. The XR device 10 then converts these screen coordinates to global coordinates. That is, the XR device 10 detects the positions of keys KH and KL in three-dimensional space relative to the origin of the global coordinate system. Detection of a MIDI signal triggers the start of the alignment process.

[0049] Thus, when the XR device 10 receives a MIDI signal corresponding to the keyboard operation of the electronic instrument 20 (an example of a signal corresponding to a user's operation on the control element of the electronic instrument), it detects the coordinates of the fingertip of user U who performed the keyboard operation (an example of a user's finger that performed the operation on the control element) as captured in the image taken by the XR device 10 (an example of an imaging device). In addition, when the XR device 10 receives a MIDI signal corresponding to the keyboard operation of the electronic instrument 20, it detects the fingertip of user U who performed the keyboard operation based on the image taken by the XR device 10.

[0050] The XR device 10 retrieves the instrument object 30 specified by the user from the library 110B (step S106). The XR device 10 may also identify the model of the electronic instrument 20 placed in front of the user U using image recognition and retrieve the instrument object 30 corresponding to the identified model from the library 110B. Alternatively, a server on the network may have a library in which the instrument objects 30 are registered. In this case, the XR device 10 can access the server and retrieve the instrument object 30 specified by the user from the server's library.

[0051] The XR device 10 aligns the instrument object 30 with the electronic instrument 20 and places it in the virtual space (step S107). In the example in Figure 3, the XR device 10 performs a modeling transformation on the instrument object 30, which is represented in the local coordinate system (for example, by moving, rotating, scaling, etc., to the coordinate CH). VAlign coordinates CH and coordinate CL V (Aligning the coordinates CL with the object) and then placing it in the global coordinate system. In other words, the XR device 10 aligns the instrument object 30 with the electronic instrument 20 and renders it in the virtual space.

[0052] In this way, the XR device 10 aligns the instrument object 30 (an example of an electronic instrument object) with the electronic instrument 20 and places it in the virtual space based on the detected coordinates (e.g., coordinates CH and CL). In other words, based on the detected coordinates, the XR device 10 places the instrument object 30 in the virtual space on the display element 170 (an example of a display that can superimpose and display an image captured by an imaging device and the virtual space) so that it corresponds to the position of the real-world electronic instrument 20 in the image. Furthermore, the XR device 10 aligns the controls included in the instrument object 30 that correspond to the controls of the real-world electronic instrument 20 that were operated by the user U (e.g., the highest key and the lowest key on the keyboard of the instrument object 30) with the coordinates (e.g., coordinates CH and CL) and places the instrument object 30 in the virtual space.

[0053] The XR device 10 generates a parallax image (step S108). That is, the XR device 10 composites the instrument object 30 onto the image corresponding to the right eye and the image corresponding to the left eye, respectively, so that the instrument object 30 is displayed at the aligned position. As a result, user U can observe an image in which the instrument object 30 is superimposed on the electronic instrument 20 in front of their eyes.

[0054] The XR device 10 returns the calibration mode to the mode it was in before the transition. This completes the process shown in Figure 5. Subsequently, if the user, for example, transitions back to calibration mode, the XR device 10 executes the processes from step S102 onwards.

[0055] In the example shown in Figure 3, when the XR device 10 receives a MIDI signal (an example of a signal corresponding to a user operation on a particular control) corresponding to a key press operation on the keys at both ends of the keyboard of the electronic instrument 20 (an example of a particular control), it detects the corresponding coordinates. The particular at least one control may be one or more specific keys other than the combination of keys KH and KL. The accuracy of the alignment of the instrument object 30 improves as the number of keys pressed increases and as the positions of the pressed keys are further apart.

[0056] It is conceivable that the instrument object 30 may be positioned at an angle in the horizontal plane relative to the electronic instrument 20 due to a misalignment of the key press positions in the depth direction relative to the two keys that are pressed. When the distance between the two keys is small, the further away they are, the greater the misalignment due to the tilt, which is the misalignment in the depth direction of the instrument object 30 relative to the electronic instrument 20 (referred to as "tilt misalignment" for convenience). In this embodiment, by pressing two keys that are far apart (for example, keys KH and KL), the tilt misalignment is suppressed across the entire keyboard. Furthermore, in this embodiment, by pressing two keys that are far apart, the spacing between each key located between the two keys (in other words, the width of each key) can be determined with high precision. Therefore, the individual keys of the electronic instrument 20 and the individual keys of the instrument object 30 are precisely aligned.

[0057] As a more specific example, in this embodiment, pressing keys KH and KL detects the distance between the two keys, as well as the lowest and highest note numbers. The difference between these note numbers allows for the determination of the number of keys, and further, dividing the distance between the two keys by the number of keys allows for the determination of the width of each key. In other words, according to this embodiment, the number of keys, the width of the keys, or both of these can be determined based on the coordinates corresponding to the keys at both ends of the electronic instrument 20. The keys at both ends are an example of multiple keys. Multiple keys are an example of at least one specific operator.

[0058] The above is the description of exemplary embodiments of the present disclosure. The embodiments of the present disclosure are not limited to those described above, and various modifications are possible within the scope of the technical idea of the present disclosure. For example, the content obtained by appropriately combining the embodiments explicitly exemplified in the specification or obvious embodiments is also included in the embodiments of the present application.

[0059] In the above embodiment, the key pressing operation on specific keys (key KH and key KL) serves as the trigger for alignment. In another embodiment, not limited to specific keys, the key pressing operation on any key or the operation on other operators may serve as the trigger for alignment.

[0060] Using FIG. 6, the case where the user U presses any three keys of the electronic musical instrument 20 during the calibration mode will be described. For convenience, the any three keys are respectively denoted as "key K1", "key K2", and "key K3". Among the keys of the keyboard included in the musical instrument object 30, the keys assigned the same note numbers as key K1 to K3 are respectively "key K1 V ", "key K2 V ", "key K3 V " respectively.

[0061] In this case, the XR device 10 identifies the keys K1 to K3 (an example of an operator) pressed by the user U from the note numbers (an example of the information included in the MIDI signal) included in each of the three input MIDI signals. The XR device 10 aligns the keys K1 V to K3 V corresponding to the identified keys K1 to K3 among the keys of the keyboard included in the musical instrument object 30 so that their coordinates match, and arranges the musical instrument object 30 in the virtual space.

[0062] In the example in Figure 6, user U can align the instrument object 30 by, for example, using appropriate keyboard operations. User U does not need to memorize keyboard operations for alignment. Therefore, user U can more easily align the instrument object 30 with the electronic instrument 20. Because the distance between keys K2 and K3 is short, there is a concern that a large tilt misalignment may occur. However, in the example in Figure 6, key K1 is also pressed. The pressing position relative to key K1 (in other words, the detected coordinates relative to key K1) corrects the tilt misalignment that may have occurred if only keys K2 and K3 were pressed.

[0063] Figure 7 illustrates the case where user U operates any control other than the keyboard of the electronic instrument 20 during calibration mode. The instrument object 30 is an object that mimics the electronic instrument 20, and not only does it have the same number of keys as the electronic instrument 20, but the layout of the various controls other than the keyboard is also the same as the electronic instrument 20. The various controls other than the keyboard include, for example, a switch panel, a rotary encoder for parameter selection, a rotary encoder for parameter value input, a pitch bend wheel, and a modulation wheel. For convenience, any control other than the keyboard will be referred to as "control 40". Among the controls included in the instrument object 30, controls that are the same as control 40 will be referred to as "control 40". V It is written as follows:

[0064] In this case, the XR device 10 identifies the control 40 pressed by user U from the information contained in the input MIDI signal (for example, the control change number such as CC001). The XR device 10 then identifies the control 40 corresponding to the identified control 40 among the controls included in the instrument object 30. V The instrument object 30 is positioned in the virtual space by aligning its coordinates with those of the electronic instrument 20. In the case of a rotary control device such as a rotary encoder, the position of the instrument object 30 may be finely adjusted by the user U rotating the rotary encoder. For example, the positional misalignment of the instrument object 30 relative to the electronic instrument 20 (e.g., misalignment in the depth direction) may be corrected in accordance with the rotational operation of the rotary encoder.

[0065] In the example in Figure 7, user U can, for example, align the instrument object 30 with an appropriate operation. User U does not need to memorize any operations for alignment. Therefore, user U can more easily align the instrument object 30 with the electronic instrument 20. For example, by pressing the keys KH and KL, as well as the operator 40 which is located at a distance in the depth direction relative to the keyboard, the misalignment of the instrument object 30 relative to the electronic instrument 20 is almost suppressed. That is, by pressing the distant keys KH and KL, the widthwise misalignment and tilt misalignment of the individual keys are suppressed, and furthermore, by pressing the operator 40, the tilt misalignment is suppressed even more. Thus, in the example in Figure 7, a specific operator includes multiple keys (e.g., keys KH and KL) and an operator other than the keys provided by the electronic instrument 20 (e.g., operator 40).

[0066] In the above embodiment, instrument objects 30 corresponding to each model are registered in library 110B. However, in another embodiment, a general-purpose instrument object may be registered in library 110B instead of the instrument objects 30 corresponding to each model. Hereinafter, the general-purpose instrument object will be referred to as a "general-purpose object".

[0067] A general-purpose object is an electronic musical instrument having a predetermined number of keys, such as 32, 44, 61, 76, or 88 keys. Various controls other than the keyboard (e.g., switch panel, rotary encoder for parameter selection, rotary encoder for parameter value input, pitch bend wheel, modulation wheel, etc.) differ for each model of the electronic musical instrument 20. In order to model a model-independent instrument object, a general-purpose object does not include controls other than the keyboard. To increase versatility, a general-purpose object may be an object consisting only of the keyboard part. A general-purpose object is an example of a second object that represents only the common parts of multiple models of the electronic musical instrument 20.

[0068] Figure 8 shows a general-purpose object 40a. General-purpose object 40a is a musical instrument object with a 44-key keyboard. In Figure 8, reference numeral 20a indicates a real-world electronic musical instrument with a 44-key keyboard.

[0069] For example, consider the case where user U presses the highest and lowest keys on the keyboard of the electronic instrument 20a (see Figure 8) during calibration mode. In this case, the XR device 10 detects the note numbers contained in the two MIDI signals received from the electronic instrument 20a and calculates the difference between the detected note numbers. Since the electronic instrument 20a has 44 keys, the difference in note numbers obtained is 43.

[0070] The XR device 10 determines that the electronic instrument 20a has 44 keys based on the calculated difference in note numbers, and calls the corresponding general-purpose object 40a. The XR device 10 estimates the width of the keys on the electronic instrument 20a, for example, by image recognition. The XR device 10 enlarges or reduces the size of the general-purpose object 40a to match the estimated width of the keys. In the example in Figure 8, the XR device 10 enlarges the general-purpose object 40a in the width direction. The XR device 10 aligns the enlarged general-purpose object 40a1 with the electronic instrument 20a and places it in the virtual space.

[0071] Figure 9 shows a general-purpose object 40b. General-purpose object 40b is a musical instrument object with a 61-key keyboard. In Figure 9, labels 20b and 20c represent real-world electronic musical instruments with a 61-key keyboard.

[0072] For example, consider the case where user U presses the highest and lowest keys on the keyboard of the electronic instrument 20b (see Figure 9) during calibration mode. In this case, the XR device 10 detects the note numbers contained in the two MIDI signals received from the electronic instrument 20b and calculates the difference between the detected note numbers. Since the electronic instrument 20b has 61 keys, the difference in note numbers obtained is 60.

[0073] The XR device 10 determines that the electronic instrument 20b has 61 keys based on the calculated difference in note numbers, and calls the corresponding general-purpose object 40b. The XR device 10 estimates the width of the keys on the electronic instrument 20b, for example, by image recognition. The XR device 10 enlarges or reduces the size of the general-purpose object 40b to match the estimated width of the keys. In the example in Figure 9, the XR device 10 reduces the width of the general-purpose object 40b. The XR device 10 aligns the enlarged general-purpose object 40b1 with the electronic instrument 20b and places it in the virtual space.

[0074] For example, consider the case where user U presses the highest and lowest keys on the keyboard of the electronic instrument 20c (see Figure 9) during calibration mode. In this case, as in the previous example, 60 is obtained as the difference in note numbers. Therefore, the XR device 10 calls the corresponding general-purpose object 40b. The XR device 10 estimates the width of the keyboard provided on the electronic instrument 20c, for example, by image recognition. The XR device 10 expands the general-purpose object 40b in the width direction to match the estimated keyboard width. The XR device 10 aligns the expanded general-purpose object 40b2 with the electronic instrument 20b and places it in the virtual space.

[0075] As shown in the examples in Figures 8 and 9, there is no need to prepare separate instrument objects for each model. Furthermore, the XR device 10 automatically detects and calls up a general-purpose object that corresponds to a real-world electronic instrument. Since the user U does not need to select an instrument object, the operational burden on the user U is reduced. The called general-purpose object is resized to match the real-world electronic instrument. Therefore, a single general-purpose object can support various models of electronic instruments with different sizes.

[0076] In the examples shown in Figures 8 and 9, both instrument objects 30 and general-purpose objects may be registered in library 110B. That is, library 110B may be an example of a storage unit that stores instrument objects 30 (an example of a first object) and general-purpose objects (an example of a second object). The XR device 10 searches library 110B (an example of a storage unit) based on the results of image recognition processing on the captured image (for example, the model identified by image recognition). If an instrument object 30 corresponding to an electronic instrument 20 in the captured image is found in the search, the XR device 10 retrieves that instrument object 30 from library 110B and places it in the virtual space. For example, if a corresponding instrument object 30 is not found in the search, the XR device 10 retrieves a general-purpose object from library 110B and places it in the virtual space. Since the number of instrument objects 30 registered in library 110B can be reduced, memory capacity can be saved. [Explanation of Symbols]

[0077] 1: System, 10: XR device, 20: Electronic instrument, 30: Instrument object, 100: Processor, 110A: Control program, 110B: Library

Claims

1. Computers When a signal corresponding to a user's operation on an electronic instrument's control element is input, the coordinates of the user's finger that performed the operation on the control element are detected based on the image captured by the imaging device. Based on the detected coordinates, on a display capable of superimposing the image captured by the imaging device and the virtual space, the objects of the electronic musical instrument are placed in the virtual space so that they correspond to the positions of the electronic musical instrument in the real world within the image. method.

2. The operator included in the object, the operator corresponding to the operator of the electronic musical instrument on which the operation was performed, is aligned to the coordinates, and the object is placed in the virtual space. The method according to claim 1.

3. When a signal corresponding to the user's operation on at least one of the electronic instruments is input, the coordinates are detected. The method according to claim 1.

4. The aforementioned specific at least one operator includes a plurality of keys provided by the electronic musical instrument, The method according to claim 3.

5. The aforementioned plurality of keys include the keys at both ends of the keyboard provided by the electronic musical instrument. The method according to claim 4.

6. Based on the coordinates corresponding to the keys at both ends, the number of keys of the electronic instrument, the width of the keys, or both are determined. The method according to claim 5.

7. The aforementioned specific at least one operator includes the plurality of keys and operators other than the keys provided by the electronic instrument, The method according to any one of claims 4 to 6.

8. The storage unit stores a first object representing a specific model of the electronic musical instrument and a second object representing only the common parts of multiple models of the electronic musical instrument. The storage unit is searched based on the results of the image recognition processing on the aforementioned image. If the first object corresponding to the electronic musical instrument in the image is not found in the search, the second object is retrieved from the storage unit and placed in the virtual space. The method according to claim 1.

9. The aforementioned signal is a MIDI (Musical Instrument Digital Interface) signal output when the control element of the electronic musical instrument is operated. From the information contained in the MIDI signal, the operator on which the operation was performed is identified, The operator included in the object, the operator corresponding to the identified operator of the electronic musical instrument, is aligned to the coordinates, and the object is placed in the virtual space. The method according to claim 1.

10. When a signal corresponding to a user's operation on an electronic instrument's control element is input, the coordinates of the user's finger that performed the operation on the control element are detected based on the image captured by the imaging device. Based on the detected coordinates, on a display capable of superimposing the image captured by the imaging device and the virtual space, the objects of the electronic musical instrument are placed in the virtual space so that they correspond to the positions of the electronic musical instrument in the real world within the image. To have the computer perform the process. program.

11. When a signal corresponding to a user's operation on an electronic instrument's control element is input, the coordinates of the user's finger that performed the operation on the control element are detected based on the image captured by the imaging device. Based on the detected coordinates, on a display capable of superimposing the image captured by the imaging device and the virtual space, the objects of the electronic musical instrument are placed in the virtual space so that they correspond to the positions of the electronic musical instrument in the real world within the image. Equipped with a control unit, Information processing device.

Images