Information processing method and information processing apparatus

Abstract

To provide an information processing method that allows a user to visually recognize the direction of a beam output from an audio device such as a speaker.SOLUTION: An information processing method according to an embodiment obtains first position information indicating the position of the ceiling surface, the wall surface, or the floor surface within a predetermined space, second position information indicating the position of an audio device that outputs an audio beam within the predetermined space, and direction information indicating the direction of an audio beam output from the audio device, calculates the trajectory of the audio beam output from the audio device on the basis of the acquired first position information, second position information, and direction information, and generates an audio beam image indicating the trajectory of the audio beam on the basis of the calculation results.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] One embodiment of this invention relates to an information processing method and an information processing apparatus.

Background Art

[0002] Patent Document 1 describes an audio processing apparatus that acquires an image of an acoustic space. The audio processing apparatus sets a plane and virtual speakers from the image of the acoustic space. The audio processing apparatus calculates a sound pressure distribution from the characteristics of the virtual speakers and generates an image in which the sound pressure distribution is superimposed on the plane.

[0003] Patent Document 2 describes a speaker device and a remote control. The speaker device measures the position of the remote control. The speaker device directs an audio beam toward the position of the remote control.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] A user cannot visually recognize the direction of an audio beam output from an acoustic device such as a speaker.

[0006] One embodiment of the present invention aims to provide an information processing method that enables a user to visually recognize the direction of an audio beam output from an acoustic device such as a speaker.

Means for Solving the Problems

[0007] An information processing method according to one embodiment of the present invention is First position information indicating the position of the ceiling, wall, or floor within a predetermined space, Second position information indicating the position of an acoustic device that emits an audio beam within a predetermined space, Directional information indicating the direction of the sound beam output from the audio equipment, Obtain, Based on the acquired first position information, second position information, and direction information, the trajectory of the sound beam output from the acoustic device is calculated. Based on the calculation results, an audio beam image showing the trajectory of the audio beam is generated. [Effects of the Invention]

[0008] According to an information processing method based on one embodiment of this invention, a user can visually recognize the direction of the sound beam output from the speaker. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a block diagram showing an example of the connection between MR goggles 1 and speaker 2. [Figure 2] Figure 2 is a block diagram showing an example of the configuration of MR goggles 1. [Figure 3] Figure 3 is a block diagram showing an example of the configuration of speaker 2. [Figure 4] Figure 4 is a perspective view showing the audio beam B1 output in space Sp. [Figure 5] Figure 5 is a plan view of space Sp. [Figure 6] Figure 6 is a perspective view showing an example of angles θ and φ of the audio beam B1 in the X', Y', and Z' axes relative to speaker 2. [Figure 7] Figure 7 shows the functional configuration of processor 13. [Figure 8] Figure 8 is a flowchart showing an example of the processing of MR goggles 1. [Figure 9] Figure 9 shows the audio beams B1 and B2 output in space Sp. [Figure 10] Figure 10 shows images of speaker 2, ceiling surface CS, wall surface WS, and floor surface FS taken with a separate camera from MR goggles 1. [Modes for carrying out the invention]

[0010] (First Embodiment) The following describes the Mixed Reality (MR) goggles 1 that implement the information processing method according to the first embodiment, with reference to the figures. Figure 1 is a block diagram showing an example of the connection between the MR goggles 1 and the speaker 2. Figure 2 is a block diagram showing an example of the configuration of the MR goggles 1. Figure 3 is a block diagram showing an example of the configuration of the speaker 2. Figure 4 is a perspective view showing the audio beam B1 output in space Sp.

[0011] MR Goggles 1 is an example of an information processing device. A user wearing MR Goggles 1 can view the real world through MR Goggles 1 while simultaneously viewing the images displayed on MR Goggles 1.

[0012] As shown in Figure 1, the MR goggles 1 are connected to speaker 2 (an example of audio equipment). Specifically, the MR goggles 1 are connected to speaker 2 wirelessly via Bluetooth® or Wi-Fi®. However, the MR goggles 1 do not necessarily have to be connected to speaker 2 wirelessly. The MR goggles 1 may also be connected to speaker 2 by a wire. In addition to speaker 2, the MR goggles 1 may also be connected to other devices (e.g., a PC, smartphone, etc.).

[0013] As shown in Figure 2, the MR goggles 1 include a communication interface 10, flash memory 11, RAM (Random Access Memory) 12, a processor 13, a display 14, and a sensor 15. The processor 13 is, for example, a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit).

[0014] The communication interface 10 is a network interface or the like. The communication interface 10 communicates with the speaker 2 wirelessly, for example, by Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like.

[0015] The flash memory 11 stores various programs. The various programs are, for example, programs for operating the MR goggles 1 and the like.

[0016] The RAM 12 temporarily stores a predetermined program stored in the flash memory 11.

[0017] The processor 13 executes various processes by reading a predetermined program stored in the flash memory 11 into the RAM 12. Note that the processor 13 does not necessarily have to execute the program stored in the flash memory 11. The processor 13 may, for example, download a program from an external device (such as a server or the like) of the MR goggles 1 via the communication interface 10 and read the downloaded program into the RAM 12.

[0018] The display 14 displays various information based on the operation of the processor 13. In the present embodiment, the display 14 of the MR goggles 1 is, for example, an organic EL display including a half mirror and a light emitting element. The user can view the display content (image or the like) reflected in the half mirror. The half mirror transmits the light incident from the front of the user. Therefore, the user can also visually recognize the real space through the half mirror.

[0019] Sensor 15 senses the environment around the MR goggles 1 and acquires it as data. In this embodiment, as shown in Figure 4, the MR goggles 1 are worn by a user in a closed space Sp consisting of a ceiling surface CS, a wall surface WS, and a floor surface FS. Sensor 15 senses positional information indicating the relative position to the ceiling surface CS, the wall surface WS, and the floor surface FS and acquires it as data. In this embodiment, sensor 15 is, for example, a stereo camera. The stereo camera acquires image data DD by photographing the area around the MR goggles 1. The stereo camera photographs the ceiling surface CS, the wall surface WS, and the floor surface FS. The stereo camera acquires image data DD in which the ceiling surface CS, the wall surface WS, and the floor surface FS have been photographed.

[0020] Furthermore, as shown in Figure 4, in this embodiment, the speaker 2 is positioned on the ceiling surface CS that constitutes the space Sp. The sensor 15 senses positional information indicating its relative position to the speaker 2 and acquires it as data. Specifically, a stereo camera, which is an example of the sensor 15, photographs the speaker 2 in addition to the ceiling surface CS, the wall surface WS, and the floor surface FS. Therefore, the stereo camera acquires image data DD in which the ceiling surface CS, the wall surface WS, the floor surface FS, and the speaker 2 are captured.

[0021] The sensor 15 does not necessarily have to be a stereo camera. The sensor 15 may be, for example, a LiDAR (Light Detection and Ranging) system. The LiDAR system measures the distance to an object by acquiring the time it takes from irradiating the object with laser light until it detects the reflected laser light from the object (speaker 2, ceiling surface CS, wall surface WS, or floor surface FS).

[0022] Speaker 2 outputs sound based on an audio signal. Speaker 2 outputs a directional audio beam B1 (see Figure 4). As shown in Figure 3, Speaker 2 includes a communication interface 20, a user interface 21, flash memory 22, RAM 23, an audio interface 24, a processor 25, multiple DA converters 26, multiple amplifiers 27, and multiple speaker units 28. In the example shown in Figure 3, only three of the multiple DA converters 26 are indicated with reference numerals. In the example shown in Figure 3, only three of the multiple amplifiers 27 are indicated with reference numerals. In the example shown in Figure 3, only three of the multiple speaker units 28 are indicated with reference numerals. The number of DA converters 26, amplifiers 27, and speaker units 28 is not limited to three, but many more.

[0023] The communication interface 20 is a network interface, etc. The communication interface 20 communicates with the MR goggles 1 wirelessly or via wired means, for example, Wi-Fi (registered trademark), Bluetooth (registered trademark).

[0024] The user interface 21 accepts various operations from the user. The user interface 21 is, for example, a remote control. The user sets the angle at which the audio beam B1 is output (the angle as seen from speaker 2) by operating the remote control (by pressing buttons, etc.).

[0025] In this embodiment, speaker 2 is positioned, for example, on the ceiling surface CS that constitutes the space Sp (see Figure 4). Speaker 2 is positioned on the ceiling surface CS such that the front surface on which the multiple speaker units 28 are arranged is parallel to the ceiling surface CS. Therefore, speaker 2 is positioned to output a sound beam B1 in the direction of the floor surface FS or the wall surface WS. For example, as shown in Figure 4, MR goggles 1 define the X, Y, and Z axes in space Sp with the position of MR goggles 1 as the reference. In this case, speaker 2 is positioned to output a sound beam B1 with respect to the -Z direction (the direction perpendicular to the ceiling surface CS and the front surface of speaker 2).

[0026] Figure 5 is a plan view of space Sp. Figure 6 is a perspective view showing an example of angles θ and φ of the sound beam B1 in the X', Y', and Z' axes relative to speaker 2. The X' direction shown in Figure 6 coincides with the -X direction shown in Figures 4 and 5. The Y' direction shown in Figure 6 coincides with the -Y direction shown in Figures 4 and 5. The Z' direction shown in Figure 6 coincides with the -Z direction shown in Figures 4 and 5. The user manually sets the angle θ in the plane of speaker 2 (angle of sound beam B1 relative to the X' direction) and the angle φ relative to the Z' direction using a remote control (user interface 21), as shown in Figures 5 and 6.

[0027] The flash memory 22 stores various programs. These various programs include, for example, a program to operate speaker 2.

[0028] RAM23 temporarily stores a predetermined program stored in flash memory22.

[0029] The audio interface 24 receives sound signals from a device other than speaker 2 via wireless or wired connections such as Wi-Fi® or Bluetooth®. This other device could be, for example, a PC or smartphone (not shown).

[0030] The processor 25 executes various processes by reading a predetermined program stored in the flash memory 22 into the RAM 23. The processor 25 is, for example, a CPU or a DSP (Digital Signal Processor). The processor 25 may consist of both a CPU and a DSP. The processor 25 does not necessarily have to execute a program stored in the flash memory 22. The processor 25 may, for example, download a program from an external device (e.g., a server) to the speaker 2 via the communication interface 20 and read the downloaded program into the RAM 23.

[0031] The processor 25 receives information indicating the direction of the audio beam B1 output from the speaker 2 (hereinafter referred to as direction information DI) in response to the operation received by the user interface 21. Specifically, the direction information DI includes angles θ and φ.

[0032] The processor 25 performs signal processing on the digital sound signal received via the audio interface 24. Signal processing includes, for example, the generation of the sound beam B1. Based on the received directional information DI, the processor 25 adjusts the delay amount so that the phase of the sound output from each of the multiple speaker units 28 aligns in a predetermined direction. In this case, the processor 25 performs delay control on the sound signal supplied to each of the multiple speaker units 28 based on the adjusted delay amount. As a result, the sound output from each of the multiple speaker units 28 reinforces each other in a predetermined direction. In other words, the processor 25 performs delay control on the sound signal supplied to each of the multiple speaker units 28 so that the sound reinforces each other in a direction (angle θ and angle φ) set by the user.

[0033] Multiple DA converters 26 receive digital audio signals that have been processed by the processor 25. The multiple DA converters 26 obtain analog audio signals by performing a DA conversion on the received digital audio signals. The multiple DA converters 26 transmit the analog audio signals to multiple amplifiers 27.

[0034] Multiple amplifiers 27 amplify the received analog sound signal. Each of the multiple amplifiers 27 transmits the amplified analog sound signal to each of the multiple speaker units 28.

[0035] Multiple speaker units 28 emit sound based on analog sound signals received from multiple amplifiers 27.

[0036] Speaker 2 does not necessarily have to accept the direction in which to output the audio beam B1 based on user operations on the user interface 21. Speaker 2 may, for example, receive information regarding the direction in which to output the audio beam B1 from a PC or smartphone (not shown) via the communication interface 20. In this case, the PC or smartphone, for example, has an application program installed for setting the direction in which to output the audio beam B1. The application program receives direction information DI in response to user operations. The application program transmits the direction information DI to speaker 2.

[0037] The following describes the process related to the visualization of the sound beam B1 in MR goggles 1 (hereinafter referred to as process P), with reference to the diagrams. Figure 7 is a diagram showing the functional configuration of processor 13. Figure 8 is a flowchart showing an example of the process in MR goggles 1.

[0038] As shown in Figure 7, the processor 13 functionally comprises an acquisition unit 130, a calculation unit 131, and a generation unit 132. The acquisition unit 130, the calculation unit 131, and the generation unit 132 execute process P.

[0039] The processor 13 starts process P, for example, when the MR goggles 1 are started up, or when a predetermined application program related to process P is executed (Figure 8: START).

[0040] After the start, the acquisition unit 130 receives image data DD from the sensor 15 (stereo camera) as shown in Figure 7 (Figure 8: Step S11).

[0041] Next, the acquisition unit 130 performs image processing (first image processing of the present invention) to recognize the ceiling surface CS, wall surface WS, or floor surface FS from the image data DD (first image data capturing the ceiling surface CS, wall surface WS, or floor surface FS) (Figure 8: Step S12). The first image processing is, for example, recognition processing by artificial intelligence such as a neural network (for example, DNN (Deep Neural Network)). The acquisition unit 130 recognizes the boundary between the ceiling surface CS and the wall surface WS, the boundary between the floor surface FS and the wall surface WS, or the boundary between two wall surfaces WS by recognition processing by artificial intelligence.

[0042] Next, the acquisition unit 130 acquires position information FLI (first position information of the present invention) indicating the position of the ceiling surface CS, wall surface WS, or floor surface FS within a predetermined space (Figure 8: Step S13). In this embodiment, the acquisition unit 130 acquires position information FLI based on the results of the first image processing. For example, the acquisition unit 130 recognizes the boundary positions of the ceiling surface CS, wall surface WS, and floor surface FS based on the images from each of the stereo cameras (two cameras). The acquisition unit 130 determines the three-dimensional coordinates of the boundary positions of the ceiling surface CS, wall surface WS, and floor surface FS based on the boundary positions of the ceiling surface CS, wall surface WS, and floor surface FS and the positional relationship between the two cameras. The acquisition unit 130 acquires position information FLI (a×x0+b×y0+c×z0=d) indicating the position of the ceiling surface CS based on the three-dimensional coordinates of the determined boundary positions. (a×x0+b×y0+c×z0=d) is a function that indicates the ceiling surface CS, which is a plane in three-dimensional space (XYZ coordinate space).

[0043] The acquisition unit 130 similarly acquires position information FLI for each surface (wall surface WS and floor surface FS). The MR goggles 1 can automatically acquire position information FLI through the first image processing.

[0044] Next, the acquisition unit 130 performs image processing (second image processing of the present invention) to recognize speaker 2 (sound equipment) from the image data DD (second image data of speaker 2) (Figure 8: step S14). The second image processing is, for example, pattern matching using template data. In this case, the MR goggles 1 have image data showing the appearance of speaker 2 etc. stored in advance as template data. The acquisition unit 130 calculates the similarity between the image data DD and the template data. The acquisition unit 130 recognizes speaker 2 when it calculates a similarity that exceeds a threshold.

[0045] Furthermore, the MR goggles 1 may recognize the speaker 2 by artificial intelligence-based object recognition processing, for example, in the same manner as the first image processing. In this case, the acquisition unit 130 recognizes the speaker 2 using a trained model that has learned the relationship between the input image and objects such as the speaker 2 through machine learning.

[0046] Next, the acquisition unit 130 acquires position information SLI (second position information) indicating the position of the speaker 2 that outputs the sound beam B1 within space Sp (within a predetermined space) (Figure 8: step S15). In this embodiment, the acquisition unit 130 acquires position information SLI based on the result of the second image processing. Specifically, when the acquisition unit 130 recognizes the speaker 2 in the second image processing, it estimates the position of the speaker 2 by image processing. The acquisition unit 130 estimates the position of the speaker 2 with the position of the MR goggles 1 as the origin. For example, in Figure 4, the acquisition unit 130 acquires the coordinates Cd1 (e.g., coordinates (x1, y1, z1), etc.) of the speaker 2 in three-dimensional space with the coordinates of the MR goggles 1 as the origin. The sensor 15 in this embodiment is a stereo camera. Therefore, the acquisition unit 130 determines the coordinate Cd1 of speaker 2 in three-dimensional space based on the position of speaker 2 recognized by the image data of each of the stereo cameras (two cameras) and the positional relationship between the two cameras. The front surface of speaker 2, where multiple speaker units 28 are arranged, is a planar mesh. Therefore, the acquisition unit 130 recognizes the planar mesh portion of speaker 2 through image processing. The acquisition unit 130 calculates the centroid position of the mesh portion and defines this centroid position as the coordinate Cd1 of speaker 2 in three-dimensional space. Note that the method of calculating the coordinate Cd1 in three-dimensional space shown above is just one example. Therefore, the acquisition unit 130 does not necessarily have to define the centroid position of the mesh-shaped portion as the coordinate Cd1 of speaker 2 in three-dimensional space. In this way, the MR goggles 1 can automatically acquire positional information SLI through the second image processing.

[0047] Next, the acquisition unit 130 acquires direction information DI indicating the direction of the audio beam B1 output from speaker 2 (Figure 8: step S16). Specifically, as shown in Figure 7, the acquisition unit 130 receives direction information DI set by the user in the user interface 21 from speaker 2.

[0048] Next, the calculation unit 131 acquires position information FLI, position information SLI, and direction information DI from the acquisition unit 130, as shown in Figure 7. Based on the acquired position information FLI, position information SLI, and direction information DI, the calculation unit 131 calculates the trajectory of the sound beam B1 output from the speaker 2 (Figure 8: Step S17).

[0049] The calculation unit 131 calculates the direction of output of the sound beam B1 within space Sp based on the direction information DI. Specifically, the calculation unit 131 obtains angles θ and φ from speaker 2 as direction information DI. Angles θ and φ are angles in a polar coordinate system with respect to the position of speaker 2. Therefore, the calculation unit 131 determines the inclination (l,m,n) of the three-dimensional Cartesian coordinate system corresponding to angles θ and φ. The calculation unit 131 defines a straight line (x,y,z)=(x1,y1,z1)+t(l,m,n) passing through the position of speaker 2 (x1,y1,z1) (t is an arbitrary value). The calculation unit 131 also determines the coordinates Cd2 of the intersection point where this straight line intersects the floor surface FS or the wall surface WS (see Figure 4). The calculation unit 131 defines the line segment from the position of speaker 2 to this intersection point as the trajectory of the sound beam B1. In other words, the calculation unit 131 defines the line segment from coordinate Cd1 to coordinate Cd2 as the trajectory of the sound beam B1.

[0050] Finally, the generation unit 132 generates an audio beam image showing the trajectory of the audio beam B1 based on the calculation results of the audio beam B1's trajectory (Figure 8: Step S18). For example, the generation unit 132 performs a calculation to associate the above 3D coordinates with the 2D coordinate positions of the display unit 14. The generation unit 132 generates an image showing the trajectory of the audio beam B1 corresponding to the calculated 2D coordinates. The generation unit 132 generates, for example, an image of a line segment having a predetermined color and a predetermined width centered on the trajectory of the audio beam B1 (such as the cylindrical image of the audio beam B1 shown in Figure 4). As a result, the generation unit 132 displays the cylindrical image as an audio beam image on the display unit 14. In this case, the user can view the audio beam image superimposed on space Sp (real space) via the display unit 14. Therefore, the user can view the audio beam image displayed on the display unit 14 while viewing the real space.

[0051] The execution of the series of processes P in the MR goggles 1 is completed by performing the processes from steps S11 to S18 described above (Figure 8: END). Note that the processor 13 may execute steps S11 to S15 after executing step S16.

[0052] (effect) In this embodiment, the MR goggles 1 display the generated sound beam image on the display unit 14. This allows the user to visually perceive the trajectory of the sound beam B1 output from the speaker 2. Therefore, the user can visually recognize the direction of the sound beam B1 output from the speaker 2. This makes it easier for the user to adjust the sound beam B1. For example, by looking at the visualized sound beam B1, the user can accurately adjust the angle of the sound beam B1. Therefore, compared to adjusting the sound beam B1 using only sound, the user can direct the sound beam B1 in the desired direction.

[0053] Furthermore, speaker 2 does not necessarily have to be placed in a closed space Sp consisting of the ceiling surface CS, wall surface WS, and floor surface FS. For example, speaker 2 may be placed in a space such as an open space that does not have a ceiling surface CS. In this case, speaker 2 may be placed, for example, on the wall surface WS or the floor surface FS.

[0054] Speaker 2 may be placed outdoors. In this case, speaker 2 is placed on the floor surface FS.

[0055] (Variation 1) The following describes the MR goggles 1a according to Modification 1 with reference to the figures. Figure 9 shows the sound beams B1 and B2 output in space Sp. As shown in Figure 9, the MR goggles 1a differ from the MR goggles 1 in that it displays an image showing the trajectory of the sound beam B2 reflected off the wall surface WS. Also, the speaker 2 in this modification differs from the embodiment in that it is located on the wall surface WS. All other configurations are the same as in the first embodiment.

[0056] Speaker 2 is positioned to output the sound beam B1 with respect to the -Y direction (the direction perpendicular to the wall surface WS and the front of speaker 2). Therefore, in this modified example, the X' direction shown in Figure 6 coincides with the -X direction shown in Figure 9. The Y' direction shown in Figure 6 coincides with the -Z direction shown in Figure 9. The Z' direction shown in Figure 6 coincides with the -Y direction shown in Figure 9. The user sets the angle θ of the sound beam B1 with respect to the X' direction and the angle φ with respect to the Z' direction of speaker 2.

[0057] The calculation unit 131 of the MR goggles 1a determines the inclination (l1,m1,n1) of the three-dimensional Cartesian coordinate system corresponding to the angles θ and φ in the polar coordinate system. The calculation unit 131 of the MR goggles 1a also determines the position (x2,y2,z2) of speaker 2 through the second image processing described above. The calculation unit 131 of the MR goggles 1a determines the coordinates Cd3 of the intersection point where the line (x,y,z)=(x2,y2,z2)+t(l1,m1,n1) passing through the position (x2,y2,z2) of speaker 2 intersects with the wall surface WS (see Figure 9). The calculation unit 131 defines the line segment from coordinate Cd1 to coordinate Cd3(x3,y3,z3) as the trajectory of the sound beam B1.

[0058] As shown in Figure 9, the sound beam B1 output from speaker 2 is reflected by the wall surface WS (coordinate Cd3). Therefore, after calculating the trajectory of sound beam B1, the calculation unit 131 calculates the trajectory of sound beam B2 reflected at coordinate Cd3. In other words, the calculation unit 131 calculates the position (coordinate Cd2) where sound beam B1 is reflected by sound beam B1 on the wall surface WS and the trajectory of sound beam B2 after reflection, based on the position information FLI, position information SLI, and direction information DI. When sound beam B1 is output in the -X direction, sound beam B2 is reflected in the X direction after being reflected by the wall surface WS. Therefore, the X-axis direction vector of the line representing sound beam B2 is inversely related to the X-axis direction vector of the line representing sound beam B1. On the other hand, the Y-axis direction vector of the line representing sound beam B2 is the same as the Y-axis direction vector of the line representing sound beam B1, and the Z-axis direction vector of the line representing sound beam B2 is the same as the Z-axis direction vector of the line representing sound beam B1. Therefore, the straight line representing the audio beam B2 is (x,y,z)=(x3,y3,z3)+t(-l1,m1,n1).

[0059] Finally, the generation unit 132 of the MR goggles 1a generates an audio beam image showing the trajectories of audio beam B1 and audio beam B2. For example, the generation unit 132 of the MR goggles 1a performs calculations to map the above 3D coordinates to the 2D coordinate positions of the display unit 14, similar to the generation unit 132 of the MR goggles 1. In this case, the audio beam image includes an image showing the trajectory of the reflected audio beam B2 (reflection image).

[0060] Furthermore, reflection is not limited to a single instance. The sound beam may be output towards the ceiling surface CS and reflected by the ceiling surface CS. Alternatively, the sound beam may be output towards the floor surface FS and reflected by the floor surface FS.

[0061] Furthermore, the MR goggles 1a may change the color of the image showing the sound beam before and after reflection based on characteristic information of the ceiling surface CS, wall surface WS, or floor surface FS (for example, the sound absorption coefficient of the ceiling surface CS, wall surface WS, or floor surface FS). Specifically, the calculation unit 131 acquires characteristic information of the ceiling surface CS, wall surface WS, or floor surface FS (for example, the sound absorption coefficient of the ceiling surface CS, wall surface WS, or floor surface FS). For example, the calculation unit 131 reads characteristic information that has been stored in the flash memory 11 in advance. The generation unit 132 changes the image showing the sound beam B2 (reflection image) based on the sound absorption coefficient. For example, the generation unit 132 makes the color of the image showing the sound beam B2 after reflection lighter than the color of the image showing the sound beam B1 before reflection, according to the sound absorption coefficient (for example, changing from dark blue to light blue).

[0062] The characteristic information is not limited to sound absorption coefficient. The characteristic information may be, for example, the hardness, roughness, thickness, or density of a surface such as a wall. In this case, the calculation unit 131 reads (acquires) characteristic information that has been stored in the flash memory 11 in advance. The generation unit 132 modifies the image based on the read characteristic information. For example, the generation unit 132 changes the intensity of the image showing the sound beam B1 according to the density of the wall, etc. (for example, changing from dark blue to light blue). Similarly, the generation unit 132 changes the intensity of the image showing the sound beam B1 based on, for example, the hardness, roughness, or thickness of a surface such as a wall.

[0063] Furthermore, the MR goggles 1a may estimate the sound absorption coefficient based on the hardness, surface roughness, thickness, or density of the acquired wall or other surface, and change the image showing the sound beam B1 based on the estimated sound absorption coefficient.

[0064] Furthermore, even if characteristic information has not been acquired, MR goggles 1a may appropriately change the color of the image showing the sound beam before and after reflection.

[0065] Furthermore, the generation unit 132 may change elements other than the color of the image representing the sound beam. For example, the generation unit 132 may change the size of the image representing the trajectory of the sound beam (for example, by changing the width of the line segment representing the sound beam) or change its shape before and after reflection.

[0066] Furthermore, the MR goggles 1a may change the sound beam image based on information other than characteristic information. For example, the generation unit 132 may change the sound beam image based on at least one of the channel of the sound beam, the volume of the sound beam, or the frequency characteristics of the sound beam. For example, the generation unit 132 may generate the sound beam image such that the color of the image of the sound beam output from the R channel of speaker 2 is different from the color of the image of the sound beam output from the L channel of speaker 2. Also, for example, the generation unit 132 may make the color darker as the volume of the sound beam increases. Also, for example, the generation unit 132 may change the color of the image representing the sound beam according to the frequency. For example, the generation unit 132 may make it red when the level of low-frequency components is high, and blue when the level of high-frequency components is high.

[0067] (effect) Because users cannot see the sound beams B1 and B2, it is extremely difficult for them to determine the direction in which sound beam B2 will be reflected off the wall. In contrast, MR goggles 1a visualize the sound beam B2 reflected off the ceiling surface CS, wall surface WS, or floor surface FS. This allows users to visually recognize the trajectory of sound beam B2 reflected off walls, etc. Therefore, it becomes easier for users to adjust the direction of sound beam B2 reflected off walls, etc.

[0068] For example, MR goggles 1a change the intensity of the color of the sound beam image before and after reflection, according to the sound absorption coefficient of the ceiling surface CS, wall surface WS, or floor surface FS. This allows the user to visually perceive changes in the volume of the sound beam B2 reflected by the wall, etc.

[0069] For example, MR goggles 1a change the audio beam image based on the channel of the audio beam. This allows the user to visually recognize, for example, whether the audio beam is being emitted from the R channel or the L channel.

[0070] For example, MR goggles 1a change the audio beam image based on the frequency characteristics of the audio beam. This allows the user to visually recognize the frequency of the audio beam.

[0071] (Modification 2) In the modified example 2, the information processing device is a VR (Virtual Reality) goggle (not shown) instead of an MR goggle. The VR goggle displays an image on the display 14 based on image data DD (camera image data) captured by the sensor 15 (stereo camera). As a result, the user of the VR goggle can perceive the real world through the image displayed on the display 14.

[0072] The VR goggles calculate the trajectory of the audio beam B1 and generate an audio beam image, similar to the processor 13 of the MR goggles 1.

[0073] The VR goggles generate an image to be displayed on the display unit 14 (hereinafter referred to as the display image) from the image data DD (camera image data), and then superimpose the audio beam image of the audio beam B1 onto the display image. The VR goggles output the display image with the audio beam image superimposed onto it to the display unit 14. This allows the user to see the trajectory of the audio beam B1 while simultaneously viewing the real space (the space around the user). In this way, the VR goggles achieve the same effect as the MR goggles 1.

[0074] Furthermore, information processing devices such as smartphones can also display images with audio beam images superimposed, similar to the above.

[0075] (Variation 3) The following describes the MR goggles 1 related to Modification 3 with reference to the figures. Figure 10 shows images of speaker 2, ceiling surface CS, wall surface WS, and floor surface FS taken with a separate camera from the MR goggles 1.

[0076] In this modified version, a camera positioned separately from MR goggles 1 (hereinafter referred to as the "shooting camera") further detects the position of user U. Specifically, the shooting camera detects position information FLI for the ceiling surface CS, wall surface WS, and floor surface FS, position information SLI for speaker 2 (sound equipment), and user position information. MR goggles 1 acquires position information FLI, position information SLI, and user position information from the shooting camera. MR goggles 1 acquires direction information DI of the sound beam from speaker 2. Based on the acquired position information FLI, position information SLI, direction information DI, and user position information, MR goggles 1 calculates the trajectory of the sound beam output from speaker 2 (sound equipment).

[0077] The camera is positioned to capture images of the user U in the MR goggles 1, speaker 2, ceiling surface CS, wall surface WS, and floor surface FS (a position where images as shown in Figure 10 can be captured). The camera acquires image data DD by capturing images of the user in the MR goggles 1, speaker 2, ceiling surface CS, wall surface WS, and floor surface FS.

[0078] The camera performs first and second image processing on the image data DD. The camera also obtains user location information from the image data DD, indicating the position of user U (coordinate Cd4 shown in Figure 10). Specifically, if the camera recognizes a person in space Sp through image processing, it estimates the position of the person in space Sp as the position of user U (coordinate Cd4). In this case, the camera obtains the coordinate Cd4 of user U with the camera's position as the origin. Similarly, the camera obtains the coordinate Cd5 of speaker 2 with the camera's position as the origin. Note that, as shown in Figure 10, if the camera recognizes MR goggles 1 through image processing, it may estimate the position of MR goggles 1 as the position of user U (coordinate Cd4). Similarly, the camera obtains location information (first location information of the present invention) FLI indicating the position of the ceiling surface CS, wall surface WS, or floor surface FS, and location information (second location information of the present invention) SLI indicating the position of speaker 2.

[0079] MR goggles 1 acquire directional information DI from speaker 2. Based on position information FLI, position information SLI, and directional information DI, MR goggles 1 calculate the trajectory of the audio beam B1. Position information FLI, position information SLI, directional information DI, and the position of user U (coordinate Cd4) are relative to the position of the camera used for imaging. Therefore, MR goggles 1 converts the position information FLI, position information SLI, and directional information DI to positions relative to coordinate Cd4 (origin), and converts the trajectory of the audio beam. MR goggles 1 displays the audio beam image. MR goggles 1 displays the audio beam image relative to the position of user U. Therefore, user U can visually recognize the direction of the audio beam B1 output from speaker 2.

[0080] (Modification 4) In the modified version 4, a first device (such as a server), different from the MR goggles 1, performs all calculations and generates the audio beam image. The MR goggles 1 (second device) in modified version 4 acquires the audio beam image generated by the server (first device) and displays the acquired audio beam image on the display unit 14.

[0081] (effect) In this modified version, instead of the MR goggles 1, a different device such as a server performs the first image processing, the second image processing, the calculation of the trajectory of the audio beam B1, and the generation of the audio beam image. Therefore, the processing load on the MR goggles 1 is reduced. As a result, even if the performance of the processor 13 in the MR goggles 1 is low, the MR goggles 1 can more easily display the audio beam image without causing delays. [Explanation of Symbols]

[0082] 1,1a...MR goggles 2...Speaker 13… Processor 130…Acquisition Department 131...Calculation section 132...Generation section DD...Image data FLI,SLI…location information DI… Directional information B1, B2... Audio beam Sp…space CS…Ceiling surface WS... Wall FS…Floor surface

Claims

1. A first position information indicating the position of a ceiling, wall, or floor in real space, Second position information indicating the position of an acoustic device that emits a sound beam within the aforementioned real space, Directional information indicating the direction of the sound beam output from the aforementioned audio equipment, Obtain, The first position information and the second position information are information indicating the relative position to the display used by the user. Based on the acquired first position information, second position information, and direction information, the trajectory of the sound beam output from the sound equipment, the position where the sound beam is reflected on the ceiling surface, the wall surface, or the floor surface, and the trajectory of the sound beam after reflection are calculated. Based on the results of the above calculation, an audio beam image showing the trajectory of the audio beam is generated. The aforementioned sound beam image includes a reflected image showing the trajectory of the sound beam after reflection. The sound beam image is displayed on the display so as to be superimposed on the real space that is visible to the user via the display. Information processing methods.

2. Characteristic information indicating the characteristics of the ceiling surface, the wall surface, or the floor surface is acquired. The reflected image is changed based on the characteristic information. The information processing method according to claim 1.

3. The aforementioned real space is a closed space composed of the ceiling, the walls, and the floor. The information processing method according to claim 1 or claim 2.

4. First image data is obtained by photographing the ceiling surface, the wall surface, or the floor surface. A first image processing is performed to recognize the ceiling surface, the wall surface, or the floor surface from the first image data. Based on the results of the first image processing, the first position information is acquired. The information processing method according to any one of claims 1 to 3.

5. A second image data of the aforementioned audio equipment is obtained, A second image processing step is performed to recognize the audio device from the second image data. Based on the results of the second image processing, the second position information is acquired. The information processing method according to any one of claims 1 to 4.

6. Acquire camera image data captured by the camera, A display image is generated from the aforementioned camera image data. The process of superimposing the audio beam image onto the display image is performed. The system outputs the display image with the aforementioned audio beam image superimposed on it. The information processing method according to any one of claims 1 to 5.

7. Obtain user location information indicating the user's location, Based on the acquired first position information, second position information, direction information, and user position information, the trajectory of the sound beam output from the sound device is calculated. The information processing method according to any one of claims 1 to 6.

8. The audio beam image is changed based on at least one of the channel of the audio beam, the volume of the audio beam, or the frequency characteristics of the audio beam. The information processing method according to any one of claims 1 to 7.

9. The first device acquires the first position information, the second position information, and the direction information, calculates the trajectory of the sound beam, and generates the sound beam image. The second device acquires the audio beam image generated by the first device and displays the acquired audio beam image on the display. The information processing method according to any one of claims 1 to 8.

10. A first position information indicating the position of a ceiling, wall, or floor surface in real space, Second position information indicating the position of an acoustic device that emits a sound beam within the aforementioned real space, Directional information indicating the direction of the sound beam output from the aforementioned audio equipment, The acquisition unit acquires the following: A calculation unit calculates, based on the acquired first position information, second position information, and direction information, the trajectory of the sound beam output from the sound device, the position where the sound beam is reflected on the ceiling surface, the wall surface, or the floor surface, and the trajectory of the sound beam after reflection. A generation unit that generates an audio beam image showing the trajectory of the audio beam based on the results of the above calculation, A display unit that displays the aforementioned audio beam image, Equipped with, The first position information and the second position information are information indicating the relative position to the display used by the user, The aforementioned sound beam image includes a reflected image showing the trajectory of the sound beam after reflection. The display device displays the audio beam image so as to be superimposed on the real space that is visible to the user through the display device. Information processing device.

11. The calculation unit further acquires characteristic information indicating the characteristics of the ceiling surface, the wall surface, or the floor surface. The generation unit changes the reflected image based on the characteristic information. The information processing apparatus according to claim 10.

12. The aforementioned real space is a closed space composed of the ceiling, the walls, and the floor. The information processing apparatus according to claim 10 or claim 11.

13. The acquisition unit is, First image data is obtained by photographing the ceiling surface, the wall surface, or the floor surface. A first image processing is performed to recognize the ceiling surface, the wall surface, or the floor surface from the first image data. Based on the results of the first image processing, the first position information is acquired. An information processing apparatus according to any one of claims 10 to 12.

14. The acquisition unit is, A second image data of the aforementioned audio equipment is obtained, A second image processing step is performed to recognize the audio device from the second image data. Based on the results of the second image processing, the second position information is acquired. An information processing apparatus according to any one of claims 10 to 13.

15. The acquisition unit acquires camera image data captured by the camera, The generating unit is A display image is generated from the aforementioned camera image data. The process of superimposing the audio beam image onto the display image is performed. The system outputs the display image with the aforementioned audio beam image superimposed on it. An information processing apparatus according to any one of claims 10 to 14.

16. The acquisition unit acquires user location information indicating the user's location, The calculation unit calculates the trajectory of the sound beam output from the sound device based on the acquired first position information, second position information, direction information, and user position information. An information processing apparatus according to any one of claims 10 to 15.

17. The generation unit changes the audio beam image based on at least one of the channel of the audio beam, the volume of the audio beam, or the frequency characteristics of the audio beam. An information processing apparatus according to any one of claims 10 to 16.

18. A device different from the aforementioned information processing device acquires the first position information, the second position information, and the direction information, calculates the trajectory of the sound beam, and generates the sound beam image. The information processing device acquires the audio beam image generated by a device different from the information processing device, The acquired audio beam image is displayed on the display unit. An information processing apparatus according to any one of claims 10 to 17.

Images