Information processing device, information processing method, program, and information recording medium

Abstract

Provided is an information processing device (101) that estimates the direction of the face of a user in the real world and outputs information corresponding to the estimation. The information processing device (101) comprises a camera (151). A detection unit (111) detects a first direction of the information processing device (101) in a first coordinate system fixed in the real world. When a photograph image photographed by the camera (151) includes a face image of the user, an estimation unit (112) estimates, from the photograph image and the face image, a second direction of the face of the user in a second coordinate system fixed with respect to the information processing device (101). A calculation unit (113) calculates a third direction of the face of the user in the first coordinate system from the detected first direction and the estimated second direction. An output unit (114) outputs information corresponding to the calculated third direction.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, a program, and an information recording medium for estimating the direction of a user's face in the real world and outputting information according to the estimation. [Background technology]

[0002] Conventionally, a technique for outputting information according to the movement of a user's head has been proposed. For example, a sound source selection device disclosed in Patent Document 1 includes: Headphones and a virtual sound source providing means for providing a plurality of localized virtual sound sources to a listener wearing the headphones via the headphones; a virtual sound source selection means for selecting one virtual sound source from the plurality of virtual sound sources; Equipped with The virtual sound source providing means a localization sound source arrangement pattern storage means for storing a plurality of localization sound source arrangement patterns of the plurality of virtual sound sources to be provided to the listener; a placement pattern selection means for selecting a desired pattern from the plurality of sound source localization patterns in response to a selection operation by the listener; a mixing means for providing the plurality of virtual sound sources according to the localization sound source arrangement pattern; having a head motion detection sensor attached to the headphones to detect a movement of the head of the listener; a head movement determination means for determining the movement of the head based on an output of the head movement detection sensor; Further comprising: The placement pattern selection means is configured to select another of the localized sound source placement patterns from the localized sound source placement pattern storage means and output the selected pattern to the mixing means when the head movement determination means detects a predetermined placement pattern changing movement from the head movement.

[0003] On the other hand, many recent smartphones and tablets use a front camera (sometimes called an in-camera, front camera, or front camera) that shoots in the same direction as the screen display to take a selfie of the user looking at the screen, and a rear camera (sometimes called a rear camera) that shoots in the opposite direction to the front camera, allowing the user to take a photo of the world unfolding in front of them while checking it on the screen.

[0004] In addition, many recent smartphones and tablets are equipped with geolocation detection functions that utilize GPS (Global Positioning System), Wi-Fi access points, Bluetooth (registered trademark) beacons, acceleration sensors, geomagnetic sensors, etc., allowing the position and orientation of the smartphone or tablet to be detected relative to a world coordinate system fixed to the real world.

[0005] Additionally, technology that provides augmented reality functions that display an expanded version of the real world on the screens of smartphones and tablets is also becoming more widespread. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 4837512 Summary of the Invention [Problem to be solved by the invention]

[0007] Here, in the technology disclosed in Patent Document 1, a head motion detection sensor provided in the headphones is used to detect the movement of the user's head.

[0008] However, while audio devices such as headphones and earphones used with smartphones and tablets that have noise canceling and external sound capture functions are becoming more common, most do not have head motion detection sensors.

[0009] Therefore, there is a strong demand for technology that can estimate the direction of a user's face using the functions of already widely used smartphones, tablets, etc.

[0010] The present invention is devised to solve the above-mentioned problems, and relates to an information processing device, an information processing method, a program, and an information recording medium for estimating the direction of a user's face in the real world and outputting information corresponding to the estimation. [Means for solving the problem]

[0011] The information processing device according to the present invention has a camera, Detecting a first orientation of the information processing device in a first coordinate system fixed to the real world; If a face image of the user is included in the captured image captured by the camera, a second orientation of the face of the user in a second coordinate system fixed to the information processing device is estimated from the captured image and the face image; calculating a third orientation of the user's face in the first coordinate system from the detected first orientation and the estimated second orientation; Information according to the calculated third orientation is output. Effect of the Invention

[0012] According to the present invention, it is possible to provide an information processing device, an information processing method, a program, and an information recording medium for estimating the direction of a user's face in the real world and outputting information according to the estimation. [Brief description of the drawings]

[0013] [Figure 1] 1 is an explanatory diagram showing a schematic configuration of an information processing device according to an embodiment of the present invention; [Diagram 2] 4 is a flowchart showing control of an information processing method executed by the information processing apparatus according to the embodiment of the present invention. [Diagram 3]1 is a photograph in place of a drawing showing a grayscale example of a display by an information processing device according to an embodiment of the present invention. [Figure 4] 1 is a photograph in place of a drawing showing a display example by an information processing device according to an embodiment of the present invention in monochrome binary. [Diagram 5] 1 is a photograph in place of a drawing showing a grayscale example of a display by an information processing device according to an embodiment of the present invention. [Figure 6] 1 is a photograph in place of a drawing showing a display example by an information processing device according to an embodiment of the present invention in monochrome binary. [Figure 7] 1 is a photograph in place of a drawing showing a grayscale example of a display by an information processing device according to an embodiment of the present invention. [Figure 8] 1 is a photograph in place of a drawing showing a display example by an information processing device according to an embodiment of the present invention in monochrome binary. [Figure 9] 1 is a photograph in place of a drawing showing a grayscale example of a display by an information processing device according to an embodiment of the present invention. [Figure 10] 1 is a photograph in place of a drawing showing a display example by an information processing device according to an embodiment of the present invention in monochrome binary. [Figure 11] 1 is a photograph in place of a drawing showing a grayscale example of a display by an information processing device according to an embodiment of the present invention. [Figure 12] 1 is a photograph in place of a drawing showing a display example by an information processing device according to an embodiment of the present invention in monochrome binary. [Figure 13] 1 is a photograph in place of a drawing showing a grayscale example of a display by an information processing device according to an embodiment of the present invention. [Figure 14] 1 is a photograph in place of a drawing showing a display example by an information processing device according to an embodiment of the present invention in monochrome binary. [Figure 15] 1 is a photograph in place of a drawing showing a grayscale example of a display by an information processing device according to an embodiment of the present invention. [Figure 16] 1 is a photograph in place of a drawing showing a display example by an information processing device according to an embodiment of the present invention in monochrome binary. [Figure 17]1 is a photograph in place of a drawing showing a grayscale example of a display by an information processing device according to an embodiment of the present invention. [Figure 18] 1 is a photograph in place of a drawing showing a display example by an information processing device according to an embodiment of the present invention in monochrome binary. [Figure 19] 11 is a photograph in place of a drawing showing an example of a display in grayscale of a stage of a virtual concert hall by an information processing device according to an embodiment of the present invention. [Figure 20] 11 is a photograph, substituted for a drawing, showing an example of a display, in monochrome binary, of a stage of a virtual concert hall by an information processing device according to an embodiment of the present invention. [Figure 21] 11 is a photograph in lieu of a drawing showing, in grayscale, a display example of a virtual room in which a plurality of displays are arranged, by an information processing device according to an embodiment of the present invention. [Figure 22] 1 is a photograph in lieu of a drawing showing, in monochrome binary, a display example of a virtual room in which a plurality of displays are arranged, by an information processing device according to an embodiment of the present invention. [Diagram 23] 11 is a drawing substitute photograph showing, in grayscale, a display example of a virtual room in which a plurality of video contents are arranged by an information processing device according to an embodiment of the present invention. [Figure 24] 11 is a drawing substitute photograph showing, in monochrome binary, a display example of a virtual room in which a plurality of video contents are arranged, by an information processing device according to an embodiment of the present invention. [Diagram 25] 11 is a drawing-substitute photograph showing, in grayscale, a display example when one video content is focused on in a virtual room by an information processing device according to an embodiment of the present invention. [Figure 26] 11 is a drawing-substitute photograph showing a display example in monochrome binary when one video content is focused on in a virtual room by an information processing device according to an embodiment of the present invention. [Figure 27] 11 is a drawing-substitute photograph showing, in grayscale, a display example when one video content is focused on in a virtual room by an information processing device according to an embodiment of the present invention. [Figure 28]11 is a drawing-substitute photograph showing a display example in monochrome binary when one video content is focused on in a virtual room by an information processing device according to an embodiment of the present invention. [Figure 29] 11 is a drawing-substitute photograph showing, in grayscale, a display example when one video content is focused on in a virtual room by an information processing device according to an embodiment of the present invention. [Diagram 30] 11 is a drawing-substitute photograph showing a display example in monochrome binary when one video content is focused on in a virtual room by an information processing device according to an embodiment of the present invention. [Diagram 31] 1 is an explanatory diagram showing a schematic configuration of an information processing device that processes an object of interest according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] An embodiment of the present invention will be described below. Note that this embodiment is for illustrative purposes only and does not limit the scope of the present invention. Therefore, a person skilled in the art can adopt an embodiment in which each or all of the elements of this embodiment are replaced with equivalents. In addition, the elements described in each example can be omitted as appropriate depending on the application. In this way, any embodiment configured according to the principles of the present invention is included in the scope of the present invention.

[0015] (composition) 1 is an explanatory diagram showing a schematic configuration of an information processing device according to an embodiment of the present invention, which will be described below with reference to the diagram.

[0016] As shown in the figure, an information processing device 101 according to this embodiment has a camera 151. It also has a detection unit 111, an estimation unit 112, a calculation unit 113, and an output unit 114. In addition, an audio device 152, a display screen 153, etc. can be used as an information output destination.

[0017] The information processing device 101 according to this embodiment is typically realized by executing a program on a portable computer such as a smartphone or a tablet. The computer is connected to various output devices and input devices, and transmits and receives information to and from these devices.

[0018] A program executed by a computer can be distributed or sold by a server to which the computer is communicatively connected, or it can be recorded on a non-transitory information recording medium such as a CD-ROM (Compact Disk Read Only Memory), flash memory, or EEPROM (Electrically Erasable Programmable ROM), and then the information recording medium can be distributed, sold, etc.

[0019] The program is installed in a non-temporary information recording medium such as a hard disk, a solid state drive, a flash memory, or an EEPROM of a computer. Then, the computer realizes the information processing device of this embodiment. In general, a central processing unit (CPU) of a computer reads a program from an information recording medium to a random access memory (RAM) under the management of an operating system (OS) of the computer, and then interprets and executes the code contained in the program. However, in an architecture in which an information recording medium can be mapped in a memory space accessible by a CPU, it may not be necessary to explicitly load a program into the RAM. Various information required in the process of executing a program can be temporarily recorded in the RAM.

[0020] Furthermore, it is desirable for the computer to be equipped with a GPU (Graphics Processing Unit) to perform various image processing calculations at high speed. By using the GPU and libraries such as TensorFlow, it becomes possible to utilize the learning and classification functions in various artificial intelligence processes under the control of the CPU.

[0021] It should be noted that the information processing device 101 of this embodiment can be configured using a dedicated electronic circuit, rather than using a computer on which software is installed. For example, a portable camera, a portable electronic game device, or the like can be used as the information processing device 101.

[0022] In this aspect, the program can also be used as material for generating wiring diagrams, timing charts, etc. of electronic circuits. In such an aspect, an electronic circuit that satisfies the specifications defined in the program is configured with an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the electronic circuit functions as a dedicated device that performs the functions defined in the program, thereby realizing the information processing device of this embodiment.

[0023] For ease of understanding, the following description will be given on the assumption that the information processing device 101 is realized by a computer executing a program.

[0024] In addition, audio devices 152 such as headphones, earphones, neck speakers, bone conduction speakers, hearing aids, etc. can be connected wirelessly or by wire to the information processing device 101 as an information output destination. It is preferable that these audio devices 152 have an external sound capture function.

[0025] As described above, the screen 153 of the information processing device 101, such as a liquid crystal display, an organic electro-luminescence (EL) display, or a paper display using electronic ink, can be used as an information output destination. By using these displays as touch screens, they can also function as input devices for the information processing device 101.

[0026] In the information processing device 101 of this embodiment, the detection unit 111 detects a first orientation of the information processing device 101 in a first coordinate system fixed to the real world.

[0027] The orientation (first orientation) of the information processing device 101 in the first coordinate system fixed to the real world can be detected via a geomagnetic sensor, an inertial sensor that detects gravity, an acceleration sensor, a gyro sensor, or the like that the information processing device 101 has.

[0028] Furthermore, the position (first position) of the information processing device 101 in the first coordinate system can also be detected by a geolocation detection function using a GPS, a Wifi access point, a Bluetooth beacon, or the like.

[0029] On the other hand, if the captured image captured by the camera 151 includes an image of the user's face, the estimation unit 112 estimates a second orientation of the user's face in a second coordinate system fixed to the information processing device 101 from the captured image and the facial image.

[0030] That is, the information processing device 101 extracts a face image drawn in a captured image by image recognition, recognizes characteristic parts such as the eyes, nose, and mouth, and then estimates the direction of the user's face relative to the information processing device 101 (second direction) based on the face image. A general face tracking technique can be applied to this process.

[0031] Note that the position (second position) of the user's face relative to the information processing device 101 may further be estimated based on the position and size of the face image in the captured image.

[0032] Furthermore, the calculation unit 113 calculates a third orientation of the user's face in the real world (first coordinate system) from the detected first orientation and the estimated second orientation.

[0033] Conversion of direction between the first coordinate system and the second coordinate system can be uniquely determined based on the first orientation. Furthermore, when the first position is detected, conversion of coordinate values ​​between the first coordinate system and the second coordinate system can be uniquely determined based on the first orientation and the first position.

[0034] Therefore, by converting the components in the second coordinate system of the second orientation estimated based on the captured image into components in the first coordinate system, it is possible to calculate a third orientation that indicates in which direction the user's face is facing relative to the world (Earth).

[0035] Then, the output unit 114 outputs information according to the calculated third direction. The information may be output to an audio device 152 worn by the user or a display screen 153.

[0036] The output information can be audio information that is mixed by setting one or more virtual sound sources in the real world and changing the intensity, tone, phase, etc. of the waveform associated with each virtual sound source according to the third orientation.

[0037] The virtual sound source may be virtually placed at a sufficiently far distance in the real world, and only the virtual direction (virtual direction) from the listening point may be determined in advance to correspond to the virtual sound source. Alternatively, the virtual sound source may be virtually placed at a position in the real world.

[0038] In the former case, the output unit 114 A virtual orientation associated with the virtual sound source; The calculated third orientation, The virtual sound source is mixed with an intensity (amplification rate) according to the angle difference. If the angle difference is small, the virtual sound source is in front of the user, so by increasing the intensity of the waveform during mixing, it becomes possible to provide the user with an augmented reality sound that changes according to the direction of the face.

[0039] In addition, when there are multiple virtual sound sources, the amplification factor during mixing is adjusted so that the average sound pressure does not change significantly when the face is turned around, that is, the ratio of the amplification factors based on the angle difference for each virtual sound source is maintained and the sum of the power of the virtual sound sources is kept almost constant, making it possible to emphasize a specific virtual sound source while maintaining the impact of the entire virtual sound sources. This is called power correction.

[0040] It is also possible to make the user aware of the direction of the virtual sound source by changing the left and right amplification factors and time difference of the stereo output according to the angle difference. For example, if the virtual sound source is on the right side of the user, a simple binaural reproduction can be realized by making the amplification factor of the right side larger than that of the left side, or by setting the time difference so that the right side precedes the left side, thereby making it possible to make the user aware of the direction of the virtual sound source.

[0041] In the latter case, the virtual direction associated with the virtual sound source is a virtual position at which the virtual sound source is located in a first coordinate system; a first orientation and a first position detected by the detection unit 111; Then, the same processing as in the former case can be carried out after the calculation.

[0042] If the measurement accuracy of the first orientation and the first position and the estimation accuracy of the second position are sufficiently high, the position of the user's face in the real world (third position) may be used to calculate the virtual orientation instead of the first position. The third position can be obtained by converting the position of the face relative to the information processing device 101 (second position) obtained by face tracking into a coordinate system fixed to the information processing device 101.

[0043] In addition, the direction of the user's face may be displayed on the display screen 153 like a compass. If a display mode is adopted in which the direction of the "needle" of the "compass" changes in response to the change in the direction of the user's face when the user turns the face, the user can confirm that the present embodiment is operating properly as long as the display screen 153 is within the user's field of vision.

[0044] In addition, by correcting the waveform of the virtual sound source in terms other than intensity according to the angular difference between the virtual orientation associated with the virtual sound source and the calculated third direction, it is possible to emphasize the virtual sound source located in front of the user.

[0045] For example, it is possible to generate direct sound and reverb sound based on the waveform of a virtual sound source, and change the mixing ratio of the two depending on the angle difference. If the angle difference is small, the ratio of direct sound can be increased to make the user feel as if the virtual sound source is coming from the front at a loud volume. This is called reverberation compensation.

[0046] In addition, by determining the central frequency range of a virtual sound source in front of the user and weakening the determined central frequency range for virtual sound sources in other directions using an equalizer, it is possible to reduce frequency fringing and make the virtual sound source in front of the user stand out and hear it. This is called central frequency range correction.

[0047] In addition, for virtual sound sources in front of the speaker, saturation can be added to enhance harmonic components to create a brighter sound, making the virtual sound source in front of the speaker stand out and making it sound more appealing to the user. This is called saturation correction.

[0048] Now, when the camera 151 of the information processing device 101 is a so-called front camera, the shooting direction thereof coincides with the display direction of the screen 153 of the display, and faces the direction in which the user is assumed to be located.

[0049] Therefore, if the user is looking at the screen 153 of the information processing device 101 from the front, the user's face should be photographed by the camera 151 .

[0050] Therefore, when the user's face is not photographed by the camera 151, it is assumed that the user is not concentrating on listening to a specific virtual sound source, and adjustments to the strength of the virtual sound source, the time difference of the stereo sound, and dramatic corrections can be set to average default values.

[0051] In addition, dramatic correction can also be adjusted by the user's gestures.

[0052] For example, when a user brings his / her face closer to the display screen 153, it is possible to perform dramatic correction such as emphasizing a virtual sound source in front, assuming that the user is trying to concentrate on the front. In this embodiment, the strength of dramatic correction can be changed based on the size of the face image drawn in the captured image.

[0053] In addition, the strength of the dramatic correction can be changed according to the user's gesture. For example, a dramatic correction may be made to emphasize a virtual sound source in front of the user, assuming that the user is trying to concentrate on the front by making a gesture of listening carefully.

[0054] For example, the user's face image and hand image can be recognized from the captured image, the position of the user's face (e.g., the center of the face) and the position of the user's hand (e.g., the tip of the little finger) can be estimated, and the strength of the dramatic correction can be changed according to the distance (closeness) between the two, making it possible to easily respond to the gesture of listening carefully.

[0055] In addition, more simply, the strength of the dramatic correction may be changed depending on the distance (closeness) between the position of the user's hand image in the captured image (for example, the position of the tip of the little finger) and a representative point of the captured image (for example, the center position of the captured image or the center position of the face image, etc.).

[0056] Here, when the center position of the captured image is adopted as the representative point, even when face tracking fails and a face image cannot be recognized, if a hand image is recognized, the strength of correction can be adjusted.

[0057] In addition, when adjusting the intensity and time difference or making corrections for the performance, rather than immediately reflecting the calculated angle difference or distance as is, it is also possible to smooth the change in value by approaching the calculated value by using the average or decay average within a recent certain period of time (for example, about 100 ms).

[0058] In the above description, the user is allowed to experience augmented reality through sound, and when the user turns his or her face in the real world (if the user's face is photographed by the camera 151 of the information processing device 101), the output of the virtual sound source changes according to that direction.

[0059] Therefore, even if the environmental sound in the real world has directionality, the output of the virtual sound source will have a directionality linked to this, and even if the audio device 152 is capable of picking up external sound, or even if the external sound enters the user's ears directly, as in the case of a neck speaker, the environmental sound and virtual sound will be mixed without any inconsistencies according to the direction of the face and provided to the user, thereby providing the user with a realistic audio augmented reality.

[0060] In addition, if the information processing device 101 has a rear camera, an augmented reality image can be generated by combining an image of the real world captured by the rear camera with the appearance of a virtual object placed in the same position as the virtual sound source, and displayed on the screen 153 of the display of the information processing device 101, thereby providing the user with both visual and auditory augmented reality.

[0061] (Flow of Control) 2 is a flowchart showing the control of an information processing method executed by an information processing device according to an embodiment of the present invention. The following description will be given with reference to this figure. Note that each step of the following processing can be omitted as appropriate depending on the application mode.

[0062] When this process is started, the information processing device 101 first initializes the parameters for playing each virtual sound source with a default value (step S200), and starts the mixing playback (step S201). Various parameters can be adopted for these parameters, such as the amplification factor (which may be an amplification factor for each channel, such as left and right, or may be an overall amplification factor), the mixing ratio of direct sound and reverb sound, and the intensity of saturation, and default values ​​are set for these parameters at the beginning of the process.

[0063] After this, the playback of the virtual sound source is executed in parallel as background processing, but the mixing parameters are changed according to the direction of the user's face, etc., by the following processing.

[0064] Next, the information processing device 101 detects a first orientation (or a first position) of the information processing device 101 in the real world (first coordinate system) via a geomagnetic sensor, a gyro sensor, an acceleration sensor, or the like (step S202).

[0065] Furthermore, the information processing device 101 attempts to extract a face image of the user from the image captured by the camera 151 by image recognition (step S203).

[0066] If the attempt is successful in extracting the user's face image (step S204; Yes), the information processing device 101 estimates a second orientation (or a second position) of the user's face relative to the information processing device 101 (in a second coordinate system) based on the face image (step S205).

[0067] Then, the information processing device 101 calculates a third orientation (or third position) of the user's face in the first coordinate system by coordinate transformation based on the estimated second orientation (or second position) and the detected first orientation (or first position) (step S206).

[0068] Next, the information processing device 101 repeats the following process for each virtual sound source (step S207).

[0069] That is, the information processing device 101 acquires a virtual direction of the virtual sound source in the first coordinate system (step S208). This virtual direction may be determined in advance, or may be calculated based on the virtual position of the virtual sound source in the first coordinate system and the first position of the information processing device 101 (or the third position of the user's face).

[0070] Next, the information processing device 101 calculates new parameters for reproduction of the virtual sound source based on the angle difference between the virtual direction and the third direction (step S209). Most simply, a new amplification factor is calculated based on the angle difference, but echo compensation and saturation compensation may also be added.

[0071] In addition, the amplification factor may be further corrected according to the distance (closeness) between the virtual position of the virtual sound source and the first position (or the third position). That is, the smaller the distance, the larger the amplification factor.

[0072] After repeating the process for all virtual sound sources (step S210), the information processing device 101 further corrects new parameters for reproducing all virtual sound sources based on their mutual relationships (step S211). This correction includes, for example, a central range correction for emphasizing the front virtual sound source compared to other virtual sound sources, and a power correction for maintaining the impact of the entire virtual sound source as it is.

[0073] Then, the update process based on the new parameters is repeated for each virtual sound source (steps S212-S214) so ​​that the playback parameters of all virtual sound sources smoothly approach the new parameters (or are simply used as the new parameters), and the process returns to step S202.

[0074] On the other hand, if extraction of the user's face image fails (step S204; No), the update process based on the default values ​​to be updated is repeated for each virtual sound source (steps S215-S217) so as to bring the parameters of all virtual sound sources closer to the default values ​​(or to leave them as the default values), and then the process returns to step S202.

[0075] Although not mentioned in the above control flow, an image of the user's hand may be recognized from the captured image, and the amplification rate of the virtual sound on the front side may be changed or the intensity of saturation correction or central range correction may be changed based on the user's gesture.

[0076] (Output to the display screen) 3 to 18 are photographs in place of drawings showing examples of display by an information processing device according to an embodiment of the present invention in grayscale or monochrome binary. The following description will be given with reference to these drawings.

[0077] 3 and 4, various information is displayed on a screen 153 of a display of an information processing device 101 that is a smartphone. When the triangular play button in the lower center of the figure is tapped, playback of a song that is a virtual sound source starts.

[0078] The detection result of the direction of the user's face and the detection result of the tip of the little finger are displayed in a window at the top of screen 153. Until the user becomes accustomed to the operation, the user can check and practice gestures by looking at the detection result and holding a position where the user's face is photographed by camera 151.

[0079] You can close the window by tapping or sliding the on / off button to the left of the play button, as shown in Figures 5 and 6. You can also display the window again by tapping or sliding the same on / off button again.

[0080] Icons of musical instruments are arranged in a circle in the center of the screen 153. These represent the orientation of the virtual sound source parts arranged in the virtual space.

[0081] In this figure, the instruments are arranged at equal intervals, but they do not necessarily have to be arranged in a circle at equal intervals, and any arrangement is possible.

[0082] When you swipe on this circle, the instruments rotate around the center of the circle, allowing you to place your favorite instruments in any direction you like, as shown in Figures 7 and 8.

[0083] The object in the center of the circle is the avatar of the user operating the device, and the direction of the white arrow indicates the direction of the user's face. The instrument icon at the end of the white arrow corresponds to the virtual sound source located in front of the user.

[0084] When this process starts, the white arrow points in a default direction (for example, upwards), and if the user turns their face or moves the position of their smartphone, the direction of the white arrow changes accordingly.

[0085] Tapping on the avatar will reset the direction of the white arrow and the placement (distance) of the instruments.

[0086] In Figures 3 and 4, a sector is displayed in the direction of the white arrow. This represents the range where the amplification rate is 0.5 times or more. By making a gesture of listening, the angle of the sector changes, letting the user know which virtual sound source is being emphasized.

[0087] There are two sliders at the bottom of the screen 153. The upper slider indicates the distance to the instruments arranged in a circle in the virtual space, and the distance can be changed by moving the slider. In the arrangement shown in Figures 3 and 4, the distance is 20 meters, but in Figures 9 and 10 it is 10 meters, and in Figures 11 and 12 it is 30 meters. The distance from the avatar to the instruments shown on the screen 153 also changes depending on this distance.

[0088] The slider below is linked to the focus level, i.e. the angle of the fan shape. As mentioned above, you can change the focus level by gestures, but you can also adjust it by moving the slider directly.

[0089] Pressing the gear-shaped settings button to the right of the play button will take you to the settings form, as shown in Figures 13 and 14.

[0090] In the setting form, the master volume (default value of the mixer amplification factor) of each instrument can be set. The information processing device 101 first calculates the amplification factor used for mixing by multiplying the master volume by a multiplier according to the angle difference, and then performs correction so that the overall power becomes approximately constant.

[0091] In Figures 15 and 16, the boost mode is set. In the boost mode, when adjusting the amplification rate so that the overall power is constant, the intensity of the virtual sound source in front is doubled, making it possible to emphasize the instruments in front.

[0092] Figures 17 and 18 are examples of output when the same functions as the smartphone above are realized on a tablet.

[0093] In these images, an augmented reality image of an empty park captured by the rear camera is displayed with a video of a virtual person playing a virtual instrument superimposed on it.

[0094] On the other hand, this embodiment can also be provided for virtual reality instead of augmented reality. Figures 19 and 20 show a virtual reality in which players of virtual instruments are arranged in a circle on the stage of a virtual concert hall, and the user is provided with a virtual reality in which the user is placed in the center.

[0095] In this display example, ten players of virtual instruments are placed on the stage, and the player avatars (in this figure, avatars of three of the ten players) are shown playing the instruments on the stage, forming a video of the players' avatars as virtual objects. The sounds of the instruments being played are associated with each virtual object as virtual sound sources, and the virtual sound sources are mixed and output in the same manner as in the above embodiment.

[0096] In this manner, the user can experience being the conductor of a virtual concert.

[0097] As in the above embodiment, the user can, by using a gesture such as listening, identify the avatar of one of the multiple performers that the user is facing, i.e., the avatar located in front of the user, as the attention object representing the subject of attention.

[0098] When the user holds the information processing device 101 in front of him / her and faces the center of the screen 153, the virtual object displayed in the center of the screen 153 becomes the object of attention.

[0099] On the other hand, even if the user holds the information processing device 101 in front of the user, when the user faces not in the center of the screen 153 but in another direction such as to the right or left, the virtual object displayed in the direction in which the user faces becomes the object of attention, not the virtual object displayed in the center of the screen. In other words, the sound generating object in which the angle difference between the virtual orientation associated with the virtual sound source and the third direction is equal to or smaller than the threshold angle is specified as the object of attention.

[0100] Instead of using a gesture, a configuration may be adopted in which the face is continuously pointed at a desired virtual object, and when the time during which the face is continuously pointed exceeds a predetermined threshold time, the virtual object is identified as the object of interest.

[0101] Once a performer's avatar has been selected as the object of attention, the user may be able to change the position and orientation of the object of attention.

[0102] For example, if screen 153 is configured as a touch screen, when the touch screen is touched and a tracing operation is performed, the target object may be moved along a path of the same shape as the path of the tracing operation, which is a parallel movement of the path. In this embodiment, since the target object is specified, it is not necessary to touch the target object itself displayed on screen 153, and the tracing operation can be performed in a place other than the place where the target object is displayed on screen 153, and the position of the target object can be changed without hiding the target object with a finger.

[0103] Also, when a rotation operation is performed by touching the screen 153 with two or three fingers, the target object may be rotated by the touch angle around an axis corresponding to the number of fingers touching. In this embodiment, it is not necessary to touch the target object itself displayed on the screen 153.

[0104] It should be noted that these various operations performed on the screen 153 configured as a touch screen can also be replaced by gestures.

[0105] In the above example, an avatar of a performer playing a musical instrument is placed as a virtual object in the virtual space, but a video played with audio can also be used as a virtual object. Figures 21, 22, 23, and 24 are display examples showing how multiple virtual displays are placed in a virtual room and virtual videos are played on the virtual displays.

[0106] In these display examples, virtual moving images played on each virtual display together with sounds functioning as virtual sound sources correspond to virtual objects.

[0107] In these figures, ten virtual videos are arranged around the user in the virtual space, and the user can compare a plurality of virtual videos side by side by holding the information processing device 101 and changing his / her orientation. The virtual sound source of the virtual video toward which the user's head is facing is output with priority over the virtual sound sources of the other virtual videos.

[0108] In this embodiment, the user can compare more than 10 virtual moving images in sequence by exchanging the virtual moving images on a virtual display that is placed in a position that is not visible to the user in the virtual space.

[0109] In this embodiment, the user can view virtual moving images in sequence by rotating his or her body in real space while holding the information processing device 101.

[0110] In addition, the virtual display may be rotated around the user within the virtual space by a gesture such as changing the orientation of a finger horizontally from right to left or left to right in a short period of time, or by sliding the touch screen that constitutes screen 153 to the left or right.

[0111] This allows multiple virtual videos to be presented to the user in a manner similar to a carousel or cover flow display.

[0112] As in the above embodiment, the user can identify any of the virtual moving images displayed on screen 153 as an object of attention by turning his / her face toward that virtual moving image and making a gesture such as listening, or by continuing to turn his / her face toward that virtual moving image for a threshold time length, etc.

[0113] When a virtual video is specified as the target object, the information processing device 101 displays and plays the virtual video specified as the target object (target video) at a specified position, such as the center of the screen 153, at a specified magnification rate, and also plays a virtual sound source associated with the target object, i.e., a sound (target sound) to be played together with the virtual video, in preference to other virtual sound sources. At this time, the mixing of the sounds to be output may be such that the target sound is a specified amplification rate and the other virtual sound sources are muted, i.e., only the target sound is output and the other virtual sounds are not output.

[0114] 25 and 26 show how the virtual video drawn in the center of the screen in FIGS. 23 and 24 is specified as the object of interest, enlarged in the center of the screen, and the video and audio of the object of interest are played back.

[0115] Figures 27, 28, 29, and 30 show that the user is dancing to the audio being played, and the orientation of the information processing device 101 and the user in real space is changing, but the video being played in the center of the screen remains the object of attention.

[0116] In this embodiment, the user can cancel the designation of the object as the object of interest by making a gesture of spreading his / her hand and bringing it closer to the camera 151 of the information processing device 101, tapping the touch screen for a short period of time, or the like.

[0117] When the identification is released, the virtual moving image surrounding the user in the virtual space may be rotated around the user so that the virtual moving image whose identification has been released is positioned in the direction in which the user's head is facing. In other words, the virtual object (the virtual orientation of the virtual object) arranged in the virtual space is rotated around the virtual starting point in the virtual space so that the virtual orientation of the virtual object whose identification has been released coincides with the calculated third orientation.

[0118] According to this aspect, immediately after the identification is released, the virtual video is positioned in the direction the user is facing, and the row of virtual videos is arranged in approximately the same order and position as before, allowing the user to intuitively compare the virtual videos in order.

[0119] The information processing device 101 according to these embodiments will be summarized and described below. Fig. 31 is an explanatory diagram showing a schematic configuration of an information processing device that processes an object of interest according to an embodiment of the present invention.

[0120] 1, the information processing device 101 according to this embodiment includes an identification unit 301 and a release unit 302. The identification unit 301 and the release unit 302 acquire various pieces of information from the detection unit 111, the estimation unit 112, and the calculation unit 113, and control the output unit 114 accordingly.

[0121] As described above, in the information processing device 101 according to this embodiment, a plurality of sound producing objects are arranged in a virtual space. Each sound producing object can be, for example, a virtual object in the above embodiment, and in the above embodiment, corresponds to an avatar of a player who plays a virtual musical instrument, or a virtual display that plays a virtual video.

[0122] Each sound-producing object is associated with a virtual sound source, which in the above embodiment corresponds to a performance sound output by a virtual musical instrument or a voice reproduced together with a virtual video.

[0123] Then, the information processing device 101 displays the state of the virtual space on the screen 153. Specifically, the state of the virtual space observed from the viewpoint position and line of sight direction corresponding to the first position and the first orientation is displayed on the screen 153 with the same direction as the shooting direction of the camera 151 as the display direction.

[0124] When the position or orientation of the information processing device 101 (screen 153) is changed, or when the position or orientation of the user's head is changed, the state of the virtual world displayed on the screen 153 changes accordingly. This allows the screen 153 of the information processing device 101 to function as a "window" through which the user can peer into the virtual space.

[0125] Here, the specification unit 301 of the information processing device 101 determines whether or not the specification condition is satisfied, and performs processing according to the result.

[0126] Moreover, the release unit 302 of the information processing device 101 determines whether or not the release condition is satisfied, and performs processing accordingly.

[0127] The specifying condition is a condition for specifying one of the multiple sound generation objects as an object of attention by the user, and the canceling condition is a condition for canceling the specification of the object of attention.

[0128] In the above embodiment, the specific conditions include a gesture of listening and continuing to face a specific sound-producing object for a predetermined period of time or more, and the release conditions include a gesture of opening the hand and approaching the camera 151 or tapping the touch screen that constitutes the screen 153, but other conditions can also be used.

[0129] In the information processing device 101, when the identification unit 301 determines that the specific condition is satisfied, it identifies the sound generation object having the smallest angle difference between the virtual orientation associated with the virtual sound source and the calculated third direction as the object of attention of the user.

[0130] The attention object is an object that the user wants to pay attention to or is presumed to pay attention to among the pronunciation objects arranged in the virtual space. In the above embodiment, the pronunciation object displayed on the screen 153 and toward which the user is facing can be the attention object. That is, if the user faces the center of the screen 153, the pronunciation object displayed in the center of the screen 153 can be the attention object; if the user faces the right edge of the screen 153, the pronunciation object displayed at the right edge of the screen 153 can be the attention object; if the user faces the left edge of the screen 153, the pronunciation object displayed at the left edge of the screen 153 can be the attention object.

[0131] Now, when an object of interest is not specified, the output unit 114 mixes the virtual sound source with an intensity according to the angle difference between the virtual orientation associated with the virtual sound source and the calculated third direction. However, when an object of interest is specified, instead of outputting information according to the calculated third direction, the output unit 114 outputs the virtual sound source associated with the specified object of interest in priority to other virtual sound sources.

[0132] In the above embodiment, the sound of the virtual instrument played by the performer corresponding to the object of interest and the sound accompanying the virtual video are output in priority over other sounds. Here, "priority" includes, for example, setting the amplification factor of the virtual sound source of the object of interest to a predetermined constant and setting the amplification factors of the other virtual sound sources to zero (mute) or a small value.

[0133] Furthermore, the information processing device 101 may display the identified attention object on the screen 153 in a manner that emphasizes it more than the other sound generation objects.

[0134] In the above embodiment, it is also possible to adopt a mode in which the color of the performer corresponding to the target object is brightened, a mark is attached to the performer, etc. Also, the virtual moving image corresponding to the target object is displayed in a predetermined size in the center of the screen to be highlighted.

[0135] In the information processing device 101, when the release condition is satisfied, the release unit 302 releases the specification as the object of attention. This ends the preferential output of the virtual sound source and the highlighting on the screen 153, and the output method described above is adopted.

[0136] The position of the virtual object arranged in the virtual space can also be rotated around the viewpoint in the virtual space. That is, the information processing device 101 rotates the virtual orientation of the sound generating object arranged in the virtual space around the viewpoint based on a gesture based on a user's hand image included in a captured image or a touch operation on the screen.

[0137] Then, when a gesture is made in which the direction of a finger is changed horizontally from right to left or left to right in a short period of time, or when a right or left slide is made on the touch screen that constitutes screen 153, the moving images of the performers and virtual displays lined up around the user's viewpoint will move, allowing the user to compare their appearances in order and their sounds.

[0138] In the above embodiment, the position and orientation of the performer's avatar can be edited by pinching the avatar or touching and rotating the avatar with multiple fingers on the screen 153. That is, while the object of interest is being identified, the information processing device 101 can change the position or orientation of the object of interest in the virtual space based on a gesture based on a user's hand image included in a captured image or a touch operation on the screen.

[0139] (summary) As described above, the information processing device according to the present embodiment has a camera, a detection unit that detects a first orientation of the information processing device in a first coordinate system fixed to the real world; an estimation unit that, if a captured image captured by the camera includes a face image of the user, estimates a second orientation of the face of the user in a second coordinate system fixed to the information processing device from the captured image and the face image; a calculation unit that calculates a third orientation of the user's face in the first coordinate system from the detected first orientation and the estimated second orientation; an output unit that outputs information according to the calculated third orientation The device is configured to include the following.

[0140] In addition, in the information processing device according to this embodiment, the information processing device is connected wirelessly or by wire to an audio device worn by the user, The output unit outputs the information to the audio device. It can be configured as follows.

[0141] In addition, in the information processing device according to this embodiment, The audio device is a headphone, an earphone, a neck speaker, a bone conduction speaker, or a hearing aid capable of capturing external sound. It can be configured as follows.

[0142] In the information processing device according to the present embodiment, the output unit A virtual orientation associated with the virtual sound source; The calculated third orientation; and The virtual sound source is mixed with the sound having an intensity corresponding to the angle difference, and the sound is output as the information. It can be configured as follows.

[0143] In addition, in the information processing device according to this embodiment, The virtual orientation is predetermined. It can be configured as follows.

[0144] In addition, in the information processing device according to this embodiment, The detection unit further detects a first position of the information processing device in the first coordinate system; The virtual direction associated with the virtual sound source is a virtual position where a virtual sound source is located in the first coordinate system; the sensed first orientation and first position; It is calculated from It can be configured as follows.

[0145] In addition, in the information processing device according to this embodiment, the information processing device displays image information corresponding to the detected first position and first orientation on a screen whose display direction is the same as the shooting direction of the camera; The waveform of the virtual sound source is corrected according to the size of the face image. It can be configured as follows.

[0146] In addition, in the information processing device according to this embodiment, If the captured image includes the face image of the user and the hand image of the user, the waveform of the virtual sound source is corrected according to the distance between the face of the user and the hand of the user in the second coordinate system. It can be configured as follows.

[0147] In addition, in the information processing device according to this embodiment, If the captured image includes an image of the user's hand, the waveform of the virtual sound source is corrected according to the distance between a representative point of the captured image and the hand image. It can be configured as follows.

[0148] In addition, in the information processing device according to this embodiment, If the captured image does not include a face image, the intensity is set to a default value. It can be configured as follows.

[0149] In addition, in the information processing device according to this embodiment, the virtual sound source is associated with a sound-producing object arranged in a virtual space; The information processing device includes: displaying, on a screen whose display direction is the same as the shooting direction of the camera, a state of the virtual space observed from a viewpoint position and a line of sight direction corresponding to the detected first position and first orientation in which the sound generating object is placed; When a specific condition is satisfied, a sound generating object having a minimum angle difference between a virtual orientation associated with the virtual sound source and the calculated third direction is identified as an object of attention by the user; the output unit outputs a virtual sound source associated with the specified object of attention in preference to other virtual sound sources, instead of outputting information according to the calculated third orientation; Displaying the specified attention object on the screen in a more emphasized manner than other sound objects; When the release condition is met, the object is released from the target object. It can be configured as follows.

[0150] In addition, in the information processing device according to this embodiment, the sound object is a video that is played with a sound, While the object of interest is being identified, the information processing device displaying the object of interest at a predetermined position on the screen at a predetermined magnification; the output unit outputs a mixed sound by amplifying the virtual sound source associated with the object of interest at a predetermined amplification rate and muting other virtual sound sources; When the specification as the target object is released, the information processing device rotates a virtual orientation of the sound producing object arranged in the virtual space around the viewpoint position so that the virtual orientation of the sound producing object whose specification as the target object has been released coincides with the calculated third orientation. It can be configured as follows.

[0151] In addition, in the information processing device according to this embodiment, The information processing device rotates a virtual orientation of a sound producing object arranged in the virtual space around the viewpoint position based on a gesture based on a hand image of the user included in the captured image or a touch operation on the screen. It can be configured as follows.

[0152] In addition, in the information processing device according to this embodiment, the sound-producing object is an avatar that produces a sound; While the target object is being identified, the information processing device changes a position or a direction of the target object in the virtual space based on a gesture based on a hand image of the user included in the captured image or a touch operation on the screen. It can be configured as follows.

[0153] The information processing method according to the present embodiment is an information processing device having a camera, Detecting a first orientation of the information processing device in a first coordinate system fixed to the real world; If a face image of the user is included in the captured image captured by the camera, a second orientation of the face of the user in a second coordinate system fixed to the information processing device is estimated from the captured image and the face image; calculating a third orientation of the user's face in the first coordinate system from the detected first orientation and the estimated second orientation; Outputting information according to the calculated third direction Configure it as follows.

[0154] The program according to the present embodiment causes a computer having a camera to: a detection unit that detects a first orientation of the information processing device in a first coordinate system fixed to the real world; an estimation unit that, if a captured image captured by the camera includes a face image of the user, estimates a second orientation of the face of the user in a second coordinate system fixed to the information processing device from the captured image and the face image; a calculation unit that calculates a third orientation of the user's face in the first coordinate system from the detected first orientation and the estimated second orientation; an output unit that outputs information according to the calculated third orientation The device is configured to function as a

[0155] The program can be distributed or sold by recording it on a non-transitory computer-readable information recording medium, or via a transitory transmission medium such as a computer communication network.

[0156] The computer-readable non-transitory information recording medium according to the present embodiment is configured to record the above-mentioned program.

[0157] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. The above-described embodiments are for the purpose of explaining the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is indicated by the claims, not the embodiments. Various modifications made within the scope of the claims and the scope of the invention equivalent thereto are considered to be within the scope of the present invention. This application claims priority based on patent application No. 2021-070745, filed in Japan on Tuesday, April 20, 2021, and, to the extent permitted by the laws and regulations of the designated countries, the contents of that basic application are incorporated into this application. [Industrial Applicability]

[0158] According to the present invention, it is possible to provide an information processing device, an information processing method, a program, and an information recording medium for estimating the direction of a user's face in the real world and outputting information according to the estimation. [Explanation of symbols]

[0159] 101 Information processing device 111 Detection unit 112 Estimation Department 113 Calculation section 114 Output section 151 Camera 152 Audio equipment 153 screens 301 Specific section 302 Release section

Claims

1. An information processing device having a camera, a detection unit that detects a first orientation of the information processing device in a first coordinate system fixed to the real world; an estimation unit that, if a captured image captured by the camera includes a face image of the user, estimates a second orientation of the face of the user in a second coordinate system fixed to the information processing device from the captured image and the face image; a calculation unit that calculates a third orientation of the user's face in the first coordinate system from the detected first orientation and the estimated second orientation; an output unit that outputs information according to the calculated third orientation Equipped with The output unit is A virtual orientation associated with the virtual sound source; The calculated third orientation; and The virtual sound source is mixed with the sound having an intensity corresponding to the angle difference, and the sound is output as the information.

23. An information processing apparatus comprising:

2. the information processing device is connected wirelessly or by wire to an audio device worn by the user, The output unit outputs the information to the audio device.

2. The information processing device according to claim 1,

3. The audio device is a headphone, an earphone, a neck speaker, a bone conduction speaker, or a hearing aid capable of capturing external sound.

3. The information processing device according to claim 2,

4. The virtual orientation is predetermined.

2. The information processing device according to claim 1,

5. The detection unit further detects a first position of the information processing device in the first coordinate system; The virtual direction associated with the virtual sound source is a virtual position where a virtual sound source is located in the first coordinate system; the sensed first orientation and first position; It is calculated from 2. The information processing device according to claim 1,

6. the information processing device displays image information corresponding to the detected first position and first orientation on a screen whose display direction is the same as the shooting direction of the camera; The waveform of the virtual sound source is corrected according to the size of the face image.

6. The information processing device according to claim 5,

7. If the captured image includes the face image of the user and the hand image of the user, the waveform of the virtual sound source is corrected according to the distance between the face of the user and the hand of the user in the second coordinate system.

2. The information processing device according to claim 1,

8. If the captured image includes an image of the user's hand, the waveform of the virtual sound source is corrected according to the distance between a representative point of the captured image and the hand image.

2. The information processing device according to claim 1,

9. If the captured image does not include a face image, the intensity is set to a default value.

2. The information processing device according to claim 1,

10. the virtual sound source is associated with a sound-producing object arranged in a virtual space; The information processing device includes: displaying, on a screen in which the sound-producing object is placed and the state of the virtual space observed from a viewpoint position and a line of sight direction corresponding to the detected first position and first orientation, the display direction of which is the same as the shooting direction of the camera; When a specific condition is satisfied, a sound generating object having a minimum angle difference between a virtual orientation associated with the virtual sound source and the calculated third direction is identified as an object of attention by the user; the output unit outputs a virtual sound source associated with the specified object of attention in preference to other virtual sound sources, instead of outputting information according to the calculated third orientation; Displaying the specified attention object on the screen in a more emphasized manner than other sound objects; When the release condition is met, the object is released from the target object.

6. The information processing device according to claim 5,

11. The information processing device rotates a virtual orientation of a sound producing object arranged in the virtual space around the viewpoint position based on a gesture based on a hand image of the user included in the captured image or a touch operation on the screen.

11. The information processing device according to claim 10.

12. the sound object is a video that is played with a sound, While the object of interest is being identified, the information processing device displaying the object of interest at a predetermined position on the screen at a predetermined magnification; the output unit outputs a mixed sound by amplifying the virtual sound source associated with the object of interest at a predetermined amplification rate and muting other virtual sound sources; When the specification as the target object is released, the information processing device rotates a virtual orientation of the sound producing object arranged in the virtual space around the viewpoint position so that the virtual orientation of the sound producing object whose specification as the target object has been released coincides with the calculated third orientation.

12. The information processing device according to claim 10 or 11.

13. the sound-producing object is an avatar that produces a sound; While the target object is being identified, the information processing device changes a position or a direction of the target object in the virtual space based on a gesture based on a hand image of the user included in the captured image or a touch operation on the screen.

12. The information processing device according to claim 10 or 11.

14. An information processing device having a camera, Detecting a first orientation of the information processing device in a first coordinate system fixed to the real world; If a face image of the user is included in the captured image captured by the camera, a second orientation of the face of the user in a second coordinate system fixed to the information processing device is estimated from the captured image and the face image; calculating a third orientation of the user's face in the first coordinate system from the detected first orientation and the estimated second orientation; An information processing method for outputting information according to the calculated third orientation, The information includes: A virtual orientation associated with the virtual sound source; The calculated third orientation; and and outputting a sound obtained by mixing the virtual sound source with an intensity according to the angle difference.

23. An information processing method comprising:

15. An information processing device having a camera, a detection unit that detects a first orientation of the information processing device in a first coordinate system fixed to the real world; an estimation unit that, if a captured image captured by the camera includes a face image of the user, estimates a second orientation of the face of the user in a second coordinate system fixed to the information processing device from the captured image and the face image; a calculation unit that calculates a third orientation of the user's face in the first coordinate system from the detected first orientation and the estimated second orientation; an output unit that outputs information according to the calculated third orientation Function as a The information includes: A virtual orientation associated with the virtual sound source; The calculated third orientation; and and outputting a sound obtained by mixing the virtual sound source with an intensity according to the angle difference. A program characterized by:

16. A computer-readable non-transitory information recording medium having the program according to claim 15 recorded thereon.

Images