Information terminal and audio mixing control method for information terminal

Abstract

This information terminal comprises: a processor; a communication interface for establishing a communication connection to an acoustic sound pickup / reproduction device; and a display. The processor performs control for displaying a virtual reality image object on the display to dispose, in real space, a virtual reality object that includes the virtual reality image object and a virtual reality acoustic object, detecting, on the basis of an ambient environmental sound picked up by the acoustic sound pickup / reproduction device, a sound source direction, which is the direction of an ambient environmental sound source relative to the information terminal, determining, on the basis of the placement position of the virtual reality acoustic object and the sound source direction of the ambient environmental sound source, whether a sound emitted from the virtual reality acoustic object mixes with the ambient environmental sound, moving, when it is determined that the sound mixes with the ambient environmental sound, the placement position of the virtual reality acoustic object or the sound source direction of the ambient environmental sound, and outputting, from the acoustic sound pickup / reproduction device, a stereophonic acoustic signal which corresponds to the moved placement position of the virtual reality acoustic object or a stereophonic acoustic signal which corresponds to the moved sound source direction of the ambient environmental sound source.

Description

Information terminal and method for controlling noise pollution in information terminal

[0001] The present invention relates to an information terminal and a sound congestion control method for the information terminal, and in particular to an information terminal that plays CGR (Computer Generated Reality) content, and relates to an information terminal that has a sound congestion control process that avoids sound congestion between an acoustic object of the CGR content and ambient environmental sounds in real space, and a sound congestion control method for the information terminal.

[0002] Regarding sound interference control processing technology for closed-type sound reproduction devices, Patent Document 1 states, "Filter L imparts to the picked-up signal the HRTF of the path from the virtual speaker SPV behind the listener to the left ear. Similarly, filter R imparts to the picked-up signal the HRTF of the path from the virtual speaker SPV behind the listener to the right ear. The L-channel audio signal and R-channel audio signal input from level adjuster L and level adjuster R are mixed with a microphone pickup signal. The microphone pickup signal is processed to be localized to the position of the virtual speaker SPV behind the listener, so that the audio sound is localized in the user's head and the ambient sounds are localized in a rear position. Therefore, the user can hear unexpected sounds (for example, the sound of an emergency vehicle) from a rear position while listening to the audio sound. (Abstract excerpt)"

[0003] JP 2014-174430 A

[0004] Patent Document 1 discloses a process for changing the sound source position of ambient environmental sounds in order to avoid mixing of the acoustic signals of an external sound source with ambient environmental sounds, but does not describe the case of viewing CGR content, and in particular does not provide a method for understanding the user's viewing state and avoiding mixing of sounds that is suitable for the user's viewing state.

[0005] The present invention has been made in consideration of the above points, and an object of the present invention is to provide an information terminal that plays CGR content and is equipped with a sound-contamination control process that avoids sound contamination between the audio signal of the CGR content and the surrounding environmental sound, and a sound-contamination control method for the information terminal.

[0006] In order to solve the above problems, the present invention has the configuration described in the claims. As an example, the present invention provides an information terminal comprising a processor, a communication interface for communication connection to a sound pickup and reproduction device, and a display, wherein the processor displays virtual reality image objects on the display to place virtual reality objects including virtual reality image objects and virtual reality audio objects in real space, detects a sound source direction of an ambient environmental sound source relative to the information terminal based on ambient environmental sound picked up by the sound pickup and reproduction device, determines whether sound emitted from the virtual reality audio object and the ambient environmental sound will cause sound interference based on the placement position of the virtual reality audio object and the sound source direction of the ambient environmental sound, and, if sound interference is determined to occur, moves the placement position of the virtual reality audio object or the sound source direction of the ambient environmental sound, and performs control to output from the sound pickup and reproduction device a stereophonic signal corresponding to the placement position of the virtual reality audio object after the movement or a stereophonic signal corresponding to the sound source direction of the ambient environmental sound after the movement.

[0007] The present invention also provides a sound contamination control method for an information terminal, in which the information terminal displays virtual reality objects including virtual reality image objects and virtual reality sound objects on a display to place the virtual reality objects in real space, acquires ambient environmental sound collected by an audio pickup and reproduction device, detects a sound source direction, which is the direction of the ambient environmental sound source relative to the information terminal, based on the ambient environmental sound, determines whether the sound emitted from the virtual reality sound object will cause sound contamination with the ambient environmental sound based on the placement position of the virtual reality sound object and the sound source direction of the ambient environmental sound source, and if it determines that sound contamination will occur, moves the placement position of the virtual reality sound object or the sound source direction of the ambient environmental sound, and performs control to output a stereophonic sound signal corresponding to the placement position of the virtual reality sound object after the movement or a stereophonic sound signal corresponding to the sound source direction of the ambient environmental sound source after the movement from the audio pickup and reproduction device.

[0008] According to the present invention, it is possible to provide an information terminal that plays CGR content, which is equipped with a sound-contamination control process that avoids sound contamination between the audio signal of the CGR content and the surrounding environmental sound, and a sound-contamination control method for the information terminal. Note that other objects, configurations, and effects will be clarified in the following embodiments.

[0009] 1 is a functional block diagram of an information terminal of the present embodiment. FIG. 1 is a configuration diagram of a head-mounted display which is an information terminal of the present embodiment. FIG. 2 is a diagram illustrating an example in which the information terminal of the present embodiment is configured with an information processing device. FIG. 3 is a diagram illustrating a first example illustrating an AR object and ambient environmental sound in mixed reality. FIG. 4 is a diagram illustrating a first example illustrating sound congestion between an audio object and ambient environmental sound. FIG. 5 is a diagram illustrating a first example of avoiding sound congestion using a user's line of sight. FIG. 6 is a diagram illustrating a second example illustrating an AR object and ambient environmental sound in mixed reality. FIG. 7 is a diagram illustrating a second example of sound congestion between an audio object and ambient environmental sound. FIG. 8 is a diagram illustrating a second example of avoiding sound congestion using a user's line of sight. FIG. 9 is a first flowchart of a sound congestion control program. FIG. 10 is a diagram illustrating a first example of avoiding sound congestion by a user selection operation on a visual image. FIG. 11 is a diagram illustrating a second example of avoiding sound congestion by a user selection operation on a visual image. FIG. 12 is a second flowchart of a sound congestion control program. FIG. 13 is a diagram illustrating an example of avoiding sound congestion by moving an AR object in the user's line of sight. FIG. 14 is a third flowchart of a sound congestion control program. FIG. 15 is a diagram illustrating an example of sound congestion between a mobile communication terminal and ambient environmental sound. FIG. 16 is a diagram illustrating an example of avoiding sound congestion with a mobile communication terminal by moving a sound source of ambient environmental sound. 10 is a fourth flowchart of the sound congestion control program.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings, components having the same functions are designated by the same reference numerals, and repeated description thereof will be omitted.

[0011] BACKGROUND ART Computer-generated reality (CGR), such as virtual reality (VR), which visualizes virtual space, and mixed reality (MR), which provides images that combine real space with augmented reality (AR) objects, is being used as content for games, maintenance work, sales promotions, and the like.

[0012] Virtual reality objects (hereinafter referred to as "AR objects") that form augmented reality include virtual reality image objects (hereinafter referred to as "AR image objects") created using computer graphics or the like, virtual reality sound objects (hereinafter referred to as "AR sound objects"), and "hybrid objects" that link AR image objects and AR sound objects. In this embodiment, to distinguish between virtual reality space and real space, objects that exist in real space are referred to as "real objects," and sounds emitted from real objects, such as sirens emitted from ambulances, running sounds emitted from trains, cars, motorcycles, etc., and conversations emitted from other people, television, radio, etc., are referred to as "ambient environmental sounds." Therefore, AR sound objects and environmental sound objects are both types of sound objects.

[0013] When the sound object is a stereophonic signal having a spatial sound image position, a closed-type sound reproduction device such as headphones or earphones is used, and the ambient sound in the real space is picked up by a microphone, superimposed on the stereophonic signal of the sound object, and reproduced by the closed-type sound reproduction device.

[0014] When a user watches CGR content, the audio signal of the audio object and the ambient sound are simultaneously provided to the user. This allows the user to watch the CGR content while understanding the situation in the surrounding environment. At this time, the spatial sound image position of the audio object may overlap with the sound source direction of the ambient sound, resulting in sound mixing, making it difficult to distinguish between the two sounds. If the ambient sound is noise, it may interfere with listening to the audio signal of the audio object. If the ambient sound is a call from a third party, listening to the audio signal of the audio object may result in the user missing the audio signal.

[0015] Therefore, the following describes an embodiment of an information terminal that controls the sound mixture between an acoustic object and ambient environmental sound that occurs while viewing CGR content in such a closed-type sound reproduction device, and a sound mixture control method for the information terminal.

[0016] The sound mixing control technology according to this embodiment is expected to contribute to "9. Build resilient infrastructure, promote inclusive and sustainable industrialization, foster innovation and build resilient infrastructure" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0017] [First Embodiment] A first embodiment will be described with reference to Fig. 1 to Fig. 8. Fig. 1 is a functional block diagram of an information terminal 1 according to this embodiment, and also illustrates a sound pickup and reproduction device 2 for ease of explanation.

[0018] 1 includes a front camera 10, a distance measurement sensor 11, an eye camera 12, a motion sensor 13, an information processing unit 3, a display 31, and an input operation unit 32. The eye camera 12 detects the gaze of the user of the information terminal 1 based on an eye image captured by the eye camera 12, and therefore corresponds to a gaze detection sensor.

[0019] The information processing unit 3 includes a processor 7 , a memory 8 , a near-field communication interface 18 , and a network interface 23 , which are connected to each other via an internal bus 33 .

[0020] The information processing unit 3 functions as a camera image processing unit 14, a distance measurement processing unit 15, a gaze detection processing unit 16, a movement detection processing unit 17, an ambient environmental sound level detection unit 19, an ambient environmental sound source direction detection unit 20, an ambient environmental sound processing unit 21, a control unit 22, an AR object receiving unit 24, an AR object placement deployment processing unit 26, a placement correction processing unit 27, a head-related transfer function determination processing unit 28, and a convolution calculation processing unit 30.

[0021] The memory 8 includes an AR object data storage unit 25 and a head-related transfer function data storage unit 29 .

[0022] The sound pickup and reproduction device 2 further includes a near-field communication interface 34, a left speaker 35 (depicted as speaker L), a right speaker 36 (depicted as speaker R), a left microphone array 37 (depicted as microphone array L), and a right microphone array 38 (depicted as microphone array R). The sound pickup and reproduction device 2 is a pair of closed-type devices, left and right, where "left" means that the device is worn on the left ear side of the user, and "right" means that the device is worn on the right ear side of the user.

[0023] A user of the information terminal 1 captures an image of the real space in front of the user with the front camera 10, and the captured image is acquired by the camera image processing unit 14 as a surrounding environment image. The user then visually recognizes the surrounding environment image and recognizes real objects in the real space, or if the display 31 is a transparent display, the user visually recognizes the real space through the transparent display and recognizes the real objects. The distance measurement sensor 11 and distance measurement processing unit 15 obtain distance data from the information terminal 1 to real objects in the real space. Furthermore, the gaze camera 12 captures an image of the area around the user's eyes, and the gaze detection processing unit 16 identifies the image the user is gazing at. The motion sensor 13 is a gyro sensor or an acceleration sensor, and the motion detection processing unit 17 detects the user's movement by combining the outputs of these sensors.

[0024] The network interface 23 accesses a server that provides CGR content services via a network and receives the CGR content. The CGR content is virtual reality or mixed reality and includes one or more AR objects as its components. The AR objects are composed of AR image objects and AR sound objects, and are received and processed by the AR object receiving unit 24. When viewing the CGR content online, the AR object placement and deployment processing unit 26 places the AR objects at coordinates in real space, i.e., displays the images of the AR image objects according to the coordinates in real space and deploys them into sound signals of the AR sound objects. Alternatively, when viewing the CGR content offline, the AR object receiving unit 24 stores the received AR objects in the AR object data storage unit 25.

[0025] The image of the AR image object is generally a three-dimensional image, and is displayed in three dimensions by being superimposed on real space (sometimes referred to as the surrounding environment) on the display 31. The display 31 may be of either a video see-through type or an optical see-through type.

[0026] Furthermore, according to the placement position of the AR sound object, the head-related transfer function determination processing unit 28 refers to the head-related transfer function data storage unit 29 to obtain a head-related transfer function. The sound signal of the AR sound object is processed by the head-related transfer function and convolution calculation processing unit 30 and converted into a stereophonic signal. The stereophonic signal is sent to the sound pickup and reproduction device 2 via the near-field communication interface 18.

[0027] The sound pickup and reproduction device 2 supplies a stereophonic signal to a left speaker 35 and a right speaker 36 to provide the user with stereophonic sound. Furthermore, the sound pickup and reproduction device 2 collects real-world sounds, i.e., ambient environmental sounds, using a left microphone array 37 and a right microphone array 38, and transmits the collected sounds to the information terminal 1 via a near-field communication interface 34. The collected ambient environmental sounds are collected as stereophonic signals, enabling the direction of the source of the ambient environmental sounds to be recognized. The stereophonic signals may be collected using a microphone array with multiple microphones, or a binaural collection method that collects sounds inside the user's ears. The noise reduction unit 39 of the sound pickup and reproduction device 2 performs noise reduction processing on the sounds acquired by the left microphone array 37 and the right microphone array 38 to generate sound data. The noise reduction processing performed here may, for example, remove sounds below a certain volume or remove components outside a predetermined frequency range (low frequency, high frequency). The noise removal unit 39 may perform noise removal processing using a known technique such as spectral subtraction.

[0028] Based on the ambient environmental sound transmitted to the information terminal 1, the ambient environmental sound level detection unit 19 detects the volume of the ambient environmental sound, and the ambient environmental sound sound source direction detection unit 20 detects the sound source direction of the ambient environmental sound. Using these detection results, the ambient environmental sound processing unit 21 performs filtering processing on the ambient environmental sound. For example, by referring to the sound source direction, distant sound sources are suppressed to make nearby sound sources easier for the user to hear, making it easier to grasp the situation of the surrounding environment. Low-level sound sources are also removed by noise filtering processing to clarify the ambient environmental sound. This may be achieved, for example, by using deep learning or clustering processing. That is, the ambient environmental sound processing unit 21 classifies each sound component based on its characteristics using deep learning. After performing various filtering processes, the sound components classified by clustering processing are grouped to generate sound data of the ambient environmental sound.

[0029] Furthermore, the placement correction processing unit 27 evaluates the overlap between the sound source direction of the detected ambient environmental sound and the placement position of the AR sound object. If the possibility of sound mixing is predicted, the placement correction processing unit 27 performs a placement correction on the AR sound object or the sound source of the ambient environmental sound. The target for placement correction depends on the image the user is gazing at. When the user is gazing at the AR image object, the AR sound object is fixed and the sound source direction of the ambient environmental sound is moved so as not to interfere with the user's viewing of the AR object. Conversely, when the user's line of sight is capturing an image in real space, the placement of the AR sound object is moved. This prevents sound mixing.

[0030] The input operation unit 32 uses a touch sensor or the like and accepts user input operations required for viewing CGR content.

[0031] The control unit 22 controls the operation settings and operation timing of the blocks connected to the internal bus 33 .

[0032] 2 is a configuration diagram of a head-mounted display 1A, which is one embodiment of the information terminal 1. In FIG. 2, blocks having the same functions as those in FIG. 1 are assigned the same numbers, and duplicated explanations will be omitted. As in FIG. 1, sound pickup and playback devices 2a and 2b (which are obtained by dividing 2 in FIG. 1 into left and right halves) are also shown. The head-mounted display 1A is an example of a head-mounted display device, and a wearable device worn on the user's head, such as smart glasses or AR glasses, may be used instead of the head-mounted display 1A.

[0033] As shown in FIG. 2 , the head-mounted display 1A includes a side head holder 40a and a front holder 40b. The head-mounted display 1A further includes a display 31 held by the side head holder 40a in front of the front holder 40b, a front camera 10 and a distance sensor 11 disposed above the display 31, and a left eye gaze camera 12L and a right eye gaze camera 12R disposed at the upper left end of the display 31 (in FIG. 1 , the left eye gaze camera 12L and the right eye gaze camera 12R are collectively illustrated as eye gaze cameras 12). The information terminal 1 further includes a motion sensor 13 disposed above the front holder 40b. An input operation unit 32 is disposed on the right outer side of the side head holder 40a, and an information processing unit 3 is disposed at the rear. A user wears the head-mounted display on their head by wearing the side head holder 40a and the front holder 40b on their head.

[0034] The display 31 is a semi-transmissive display or a non-transmissive display. In the case of mixed reality, with a semi-transmissive display, the user views the real space in front of them through a semi-transmissive screen. With a non-transmissive display, the user can confirm the real space in front of them by displaying a camera image of the real space in front of them on the non-transmissive display. In the case of virtual reality, a virtual reality image is projected over the entire surface of the non-transmissive display. The display 31 may also be a three-dimensional display.

[0035] The front camera 10 is attached so as to capture an image of the real space in front of the user's line of sight, and the distance measurement sensor 11 measures the distance to a real object in the real space.

[0036] The information processing unit 3 captures camera images captured by the front camera 10, distance data measured by the distance sensor 11, and images of the user's eyes captured by the left eye camera 12L and right eye camera 12R, and processes them as described in FIG. 1 . The information processing unit 3 also incorporates a near-field communication interface 18, which transmits stereophonic signals and the like to the sound pickup and reproduction devices 2a and 2b. The user wears the sound pickup and reproduction devices 2a and 2b on their ears and listens to stereophonic sounds through the left speaker 35 and right speaker 36. The sound pickup and reproduction devices 2a and 2b collect ambient sounds as stereophonic signals using the left microphone array 37 and right microphone array 38, and transmit the signals to the information terminal 1.

[0037] FIG. 3 is a diagram showing an example in which the information terminal 1 of this embodiment is configured as an information processing device.

[0038] 3 is a diagram showing an example in which the information terminal 1 of this embodiment is configured as an information processing device 1B, and for ease of explanation, a sound pickup and reproduction device 2 is also shown. The sound pickup and reproduction device 2 is the same as the device shown in FIG. 1. The information processing device 1B is an information terminal that is a non-wearable device such as a smartphone, tablet terminal, or personal computer.

[0039] The information processing device 1B shown in Fig. 3 includes a front camera 10, a distance measurement sensor 11, an eye camera 12, a display 31, and an input operation unit 32, which are the same as the blocks shown in Fig. 1. The information processing device 1B further includes a communication interface 60, a CPU 61, a RAM 62, a flash memory 63, and a group of sensors 70, which are connected to one another by an internal bus 71.

[0040] The flash memory 63 holds a basic operation program 64 such as an operating system, a CGR viewing program 65, a sound contamination control program 66, and a data section 69. The data section 69 stores AR object data 67 and head-related transfer function data 68.

[0041] The communication interface 60 supports wireless LAN protocols as well as near field communication. The wireless LAN interface 60a is used to connect to an external network and receive CGR content from an external server, etc. The near field communication interface 60b transmits and receives stereophonic signals, etc. to and from the sound pickup and reproduction device 2.

[0042] The sensor group 70 serves as a motion sensor that detects the motion of the user, and includes a gyro sensor 70a, an acceleration sensor 70b, and a geomagnetic sensor 70c.

[0043] The CPU 61 and RAM 62 constitute a computer system. The CPU 61 loads a basic operation program 64, a CGR viewing program 65, and a sound mixture control program 66 into the RAM 62 and executes them. The data unit 69 may store processing results generated when the CPU 61 executes the programs.

[0044] 4 is a diagram showing a first example of an AR object and ambient environmental sound in mixed reality, focusing on the display screen during execution on the information terminal 1 of FIG. 1, FIG. 2, or FIG.

[0045] As shown in FIG. 4 , a user 4 wears a head-mounted display 1A on his or her head and an acoustic pickup and reproduction device 2 on his or her ears. The user 4 views a real space image 5 displayed on a display 31 of the head-mounted display 1A. The real space image 5 is an image obtained by capturing a portion of real space and corresponds to the field of view of the user 4. If the display 31 of the head-mounted display 1A is a non-transparent display, the user 4 views the real space by viewing the real space image 5. Furthermore, if smart glasses equipped with a translucent display are used instead of the head-mounted display 1A, the user 4 views the real space directly through the translucent display and views various AR objects displayed on the translucent display superimposed on the real space.

[0046] The line of sight 6 of the user 4 is on an AR image object 51a that constitutes a hybrid object of a train placed on the real space image 5. An AR sound object 51b is also placed at the position of the AR image object 51a. An airplane hybrid object is also placed on the real space image 5, and an AR image object 52a and an AR sound object 52b are placed in the same position.

[0047] Furthermore, a real object 50a made of a car is traveling near the position where the user 4 is located in real space, and the car's running sound is being generated. Therefore, the real object 50a is also a surrounding environmental sound source 50b.

[0048] In Figure 4, the AR sound objects 51b and 52b are located in a different spatial position from the ambient environmental sound source 50b, and the direction of the sound source of the ambient environmental sound source 50 relative to the user 4 is different from the spatial position directions of the AR sound objects 51b and 52b, so no sound congestion occurs.

[0049] FIG. 5A is a diagram illustrating a first example of a sound mixture of an acoustic object and ambient environmental sound in mixed reality, similar to FIG. 4 .

[0050] The ambient environmental sound source 50b and the AR sound object 51b are located in approximately the same direction as seen from the user 4, resulting in sound mixture. In this case, the information terminal 1 (head-mounted display 1A) executes a sound mixture control method characteristic of this embodiment.

[0051] FIG. 5B is a diagram showing a first example of avoiding the sound congestion in the example shown in FIG. 5A where sound congestion has occurred, by using the user's line of sight.

[0052] When the user's line of sight 6 is fixed on the AR image object 51a, if it is considered that the AR sound object 51b and the ambient environmental sound source 50b are in almost the same direction as seen by the user 4 and sound contamination will occur, priority is given to the user's line of sight 6 being fixed on the AR image object 51a, and the ambient environmental sound source 50b that is not the object of gaze is moved away from the position of the AR sound object 51b. Specifically, sound contamination control processing is executed so that the user 4 can hear the sound of a car traveling from a position 53 after the ambient environmental sound source 50b that was close to the AR sound object 51b has moved away from the AR sound object 51b. Details of the sound contamination control processing will be described later with reference to FIG. 8 .

[0053] FIG. 6 is a diagram showing a second example of an AR object and ambient environmental sounds in mixed reality.

[0054] In Figure 6, the same reference numerals as in Figure 4 denote the same things, and explanations may be omitted. In Figure 6, a third party real object 54a is emitting a voice, and this voice is an ambient environmental sound source 54b. In response to the ambient environmental sound source 54b being called by the third party real object 54a, the user 4 turns his line of sight 6 toward the third party real object 54a. In Figure 6, the direction of the sound source of the ambient environmental sound source 54b relative to the user 4 differs from the spatial position directions of the AR sound objects 51b and 52b, so no sound congestion occurs.

[0055] FIG. 7A is a diagram illustrating a second example illustrating a sound mixture of an acoustic object and a surrounding environmental sound in mixed reality, similar to FIG. 6 .

[0056] 6, the real object 54a, which is a third party, moves, and the ambient environmental sound source 54b and the AR sound object 51b are in almost the same direction as seen by the user 4, resulting in sound congestion. In this case, the information terminal 1 (head-mounted display 1A) executes a sound congestion control method that is characteristic of this embodiment.

[0057] FIG. 7B is a diagram showing a second example of how to avoid the sound congestion in the situation where the sound congestion shown in FIG. 7A occurs, by using the user's line of sight.

[0058] In a situation where the user's line of sight 6 is directed around the position of the real object 54a consisting of a third party as shown in Figure 6, and the AR sound object 51b and the ambient environmental sound source 50b are directed in substantially the same direction, causing sound congestion, the entire hybrid AR object consisting of the AR image object 51a and the AR sound object 51b is moved because the user's line of sight 6 immediately before is not on the AR image object 51a. Figure 7B shows the AR image object 51a (53) and the AR sound object 51b (53) moved to a post-movement point 53 away from the real object 54a and the ambient environmental sound source 54b. This not only eliminates sound congestion by moving the ambient environmental sound source 50b consisting of the voice of a third party away from the AR sound object 51b, but also makes the AR image object 51a, which was covering the real object 54a consisting of a third party, visible to the third party because the AR image object 51a moves to the post-movement point 53.

[0059] 8 is a first flowchart of the sound mixture control program 66. The sound mixture control program 66 operates in parallel with the processing for viewing CGR content by the CGR viewing program 65.

[0060] S10: The sound mixture control program 66 and the CGR viewing program 65 start their programs.

[0061] S11: The AR object layout processing unit 26 obtains layout position information of the AR sound object from the CGR viewing program 65.

[0062] S12: The ambient sound source direction detection unit 20 detects the sound source direction of the ambient sound.

[0063] S13: The placement correction processing unit 27 evaluates the possibility of sound mixing. If it is determined that there is a possibility of sound mixing (S13: YES), the process proceeds to S14, and if it is determined that there is no possibility of sound mixing (S13: NO), the process proceeds to S19.

[0064] S14, S15: The gaze detection processing unit 16 detects the user's gaze (S14), and the placement correction processing unit 27 determines the gaze image the user is gazing at based on the real space image 5 and the user's gaze (S15). If the placement correction processing unit 27 determines that the target of the user's gaze is an AR image object (S15: AR image object), the process proceeds to S16. If the placement correction processing unit 27 determines that the target of the user's gaze is real space or a real object (S15: real space), the process proceeds to S18. Note that the process may be configured to proceed to S19 if the user's gaze is neither on the AR image object nor in real space.

[0065] S16: If there is a possibility of sound contamination in S13 (S13: YES) and it is determined in S15 that the target of the user's gaze is the AR image object (S15: AR image object), in S16 the head-related transfer function determination processing unit 28 selects a head-related transfer function to apply to the ambient environmental sound source.

[0066] S17: The convolution calculation processing unit 30 performs a convolution calculation using the head-related transfer function applied to the ambient environmental sound source, and obtains the ambient environmental sound that has moved in the sound source direction of the ambient environmental sound source.

[0067] S18: On the other hand, if there is a possibility of sound congestion in S13 (S13: YES) and it is determined in S15 that the target of the user's gaze is real space, a real object, or an image thereof, the placement correction processing unit 27 moves the placement position of the AR sound object. Therefore, if the user is viewing an AR image object, a convolution operation is performed on the ambient environmental sound source, or if the user is viewing real space, the placement position of the AR sound object is moved.

[0068] S19: The head-related transfer function determination processing unit 28 selects a head-related transfer function corresponding to the arrangement position of the AR sound object. Note that if there is no possibility of sound congestion in S13, or if there is a possibility of sound congestion but the user 4 is viewing an image in real space, the sound source of the ambient sound is not moved.

[0069] S20: The convolution calculation processing unit 30 performs a convolution calculation using a head-related transfer function corresponding to the arrangement position of the AR sound object selected in S19, and obtains a stereophonic signal of the AR sound object.

[0070] S21: The control unit 22 superimposes the ambient environmental sound and the stereophonic signal of the sound object and outputs the superimposed signal.

[0071] S22: The control unit 22 checks the termination condition, and if it is to be terminated (S22: YES), the process ends in S23, and if it is not to be terminated (S22: NO), the process returns to before S11.

[0072] As described above, according to the first embodiment, it is possible to provide an information terminal that plays CGR content, an information terminal that is equipped with a sound-contamination control process that avoids sound mixing between the audio signal of the CGR content and the surrounding environmental sound based on the user's line of sight, and a sound-contamination control method for an information terminal.

[0073] [Second Embodiment] A second embodiment will be described with reference to Figures 9 to 11. In the second embodiment, the information terminals described in Figures 1, 2, and 3 can also be applied, and in order to reduce costs, it is possible to omit the eye camera 12 and the eye gaze detection processing unit 16. In the second embodiment, the user 4 selects an image to be viewed using the input operation unit 32 of the information terminal 1. The input operation unit 32 may be a touch panel that can be used in conjunction with other operations, or a dedicated operation means for selecting an image to be viewed may be provided.

[0074] 9 is a diagram showing a first example of avoiding sound congestion by a user selection operation of a visual recognition image. The same components as those in FIG. 4 are assigned the same numbers, and redundant explanations may be omitted.

[0075] 9, the user 4 selects an AR image object as the viewing target. At this time, mosaic processing or blurring processing is performed on the real space image 5 that is not the viewing target, thereby emphasizing the AR image objects 51 a and 52 a. As shown in FIG. 7A, when the ambient environmental sound source 50 b overlaps with the AR audio object 51 b, the ambient environmental sound source 50 b is moved to a post-movement point 53 to prevent sound congestion.

[0076] 10 is a diagram showing a second example of avoiding sound congestion by a user selection operation of a visual recognition image. The same components as those in FIG. 6 are assigned the same numbers, and duplicated explanations may be omitted.

[0077] In Fig. 10, a user 4 selects a real space image 5 to be viewed. At this time, mosaic processing and blurring processing 56a and 56b are applied to AR image objects 52a and 55a that are not to be viewed, thereby emphasizing the real space image 5. In Fig. 10, a real object 54a consisting of a third party exists in the real space. As in Fig. 7A, when an ambient environmental sound source 54b overlaps with an AR sound object 51b, the AR objects are moved to the positions of AR image object 55a and AR sound object 55b, thereby preventing sound congestion.

[0078] Fig. 11 is a second flowchart of the sound-confusion control program 66 corresponding to the second embodiment. In the second flowchart of the sound-confusion control program 66 in Fig. 11, the same steps as those in the first flowchart of the sound-confusion control program 66 shown in Fig. 8 are given the same numbers, and their explanations will be omitted.

[0079] 11, the second flowchart of the sound contamination control program 66 involves selecting an image to be viewed by the user using the input operation unit 32. The difference from the first flowchart of the sound contamination control program 66 in Fig. 8 is that the step of image selection in S30 replaces the step of specifying the gaze image by gaze detection in S14 and S15.

[0080] That is, in the second flowchart of FIG. 11, the processes from S10 to S13 are executed in the same manner as in the first flowchart of FIG. 8, and the possibility of occurrence of sound congestion is determined.

[0081] S30: If it is determined in S13 that there is a possibility of sound congestion (S13: YES), it is confirmed in S30 whether the image the user is gazing at has been selected by user operation. If there is no user operation in S30 (S30: no selection), it is determined that the user is not concerned about the possibility of sound congestion, and stereophonic signals of the AR sound objects are generated in S19 and S20.

[0082] If an AR image object is selected in S30 (S30: AR image object), similar to the first flowchart of Figure 8, a head-related transfer function is selected in S16 to move the sound source of the ambient environmental sound, and a convolution operation using this is performed in S17 to avoid sound congestion.

[0083] Furthermore, if the real space is selected in S30 (S30: real space), the AR sound object is moved in S18 to avoid sound congestion, similarly to the first flowchart of FIG.

[0084] As described above, according to the second embodiment, it is possible to provide an information terminal that plays CGR content, an information terminal that has a sound contamination control process that avoids sound contamination between the audio signal of the CGR content and the surrounding environmental sound by the user's gaze image selection operation without requiring detection of the user's gaze, and a sound contamination control method for the information terminal. Therefore, for example, if a smartphone is used as the information processing device 1B, sound contamination control can be performed by capturing a real space image 5 with the smartphone's front camera 10, displaying it on the display 31, and selecting a gaze image from a touch panel (corresponding to the input operation unit 32) stacked on the display 31.

[0085] [Third Embodiment] A third embodiment will be described with reference to Figures 12 and 13. In the third embodiment as well, the information terminals described with reference to Figures 1, 2, and 3 can be applied.

[0086] 12 is a diagram showing an example of avoiding sound congestion by moving an AR object in the user's line of sight. In FIG. 12, the same components as those in FIG. 4 are assigned the same numbers, and redundant explanations may be omitted.

[0087] In the first embodiment, when the ambient environmental sound source 50b overlaps with the AR sound object 51b as in Fig. 5A, the ambient environmental sound source 50b is moved as in Fig. 5B, but in this third embodiment, sound congestion is avoided by moving the AR object in the line of sight of the user 4. The AR object is moved from the position of the line of sight 6 to the position of the line of sight 6a, i.e., the AR image object 57a and the AR sound object 57b, at a speed that induces the movement of the line of sight, thereby preventing sound congestion without the user having to take their line of sight away from the target they are viewing.

[0088] Fig. 13 is a third flowchart of the sound-confusion control program 66 corresponding to the third embodiment. In the third flowchart of the sound-confusion control program 66 in Fig. 13, the same steps as those in the first flowchart of the sound-confusion control program 66 shown in Fig. 8 are given the same numbers, and their explanations will be omitted.

[0089] The third flowchart of the sound contamination control program 66 in FIG. 13 differs from the first flowchart of the sound contamination control program 66 in FIG. 8 in that S40 and S41 are newly added instead of S15, S16, and S17, and the execution order of S14 is changed.

[0090] That is, first, in the third flowchart of FIG. 13, similarly to the first flowchart of FIG. 8, the processes from S10 to S13 are executed to determine the possibility of occurrence of sound congestion.

[0091] S40: If it is determined in S13 that there is a possibility of sound congestion (S13: YES), the AR image object is gradually moved in S40.

[0092] S14: Then, in S14, the user's line of sight is detected.

[0093] S41: Determine whether the user's gaze is following the movement of the AR image object's display position on the display 31. If the following determination is NG (S41: NG), wait for the gaze to follow. If gaze following is confirmed (S41: OK), return to S13 and evaluate sound congestion. Repeat the processes of S13, S40, S14, and S41 until the sound congestion is resolved (S13: YES). If the sound congestion is resolved (S13: NO), select a head-related transfer function of the AR sound object according to the placement position of the AR image object (S19), generate a stereophonic signal by convolution (S20), and obtain a stereophonic signal of the AR sound object with the sound congestion resolved. Then, superimpose the generated stereophonic signal on the surrounding environmental sound and output (S21).

[0094] As described above, according to the third embodiment, it is possible to provide an information terminal that plays CGR content, an information terminal that is equipped with a sound-contamination control process that avoids sound contamination between the acoustic signal of the CGR content and the ambient environmental sound, without requiring a process to move the sound source of the ambient environmental sound, and a sound-contamination control method for an information terminal.

[0095] [Fourth Embodiment] A fourth embodiment will be described with reference to Figures 14A, 14B, and 15. The information terminals described with reference to Figures 1, 2, and 3 are also applicable to the fourth embodiment. In the fourth embodiment, a user is making a call using a mobile communication terminal, and the mobile communication terminal is placed in real space as an AR object.

[0096] Fig. 14A is a diagram illustrating an example of sound mixing between a mobile communication terminal and a surrounding environmental sound. In Fig. 14A, the same components as in Fig. 4 are assigned the same numbers, and duplicated explanations may be omitted. Fig. 14B is a diagram illustrating an example of avoiding sound mixing with a mobile communication terminal by moving the sound source of the surrounding environmental sound.

[0097] 14A shows an AR image object 58a of the mobile information terminal and an AR sound object 58b of a call heard from the mobile communication terminal. When user 4 holds the mobile communication terminal in his / her hand or places it near user 4 to make a call, the AR object of the mobile communication terminal is placed in real space. This allows user 4 to operate the mobile communication terminal and make a call without taking a picture of the mobile communication terminal with front camera 10. Note that the mobile information terminal may be other than a mobile communication terminal, such as a tablet PC or notebook PC.

[0098] Call audio data between the information terminal 1 and the mobile communication terminal is transmitted and received via the near field communication interface 18, and the call audio is reproduced as an AR sound object, for example, as a stereophonic signal from in front of the user. In Fig. 14A, the AR sound object 58b of the mobile communication terminal and the ambient sound source 50b are close to each other, causing sound congestion.

[0099] 14B, the ambient environmental sound source 50b is moved to avoid sound interference with the mobile communication terminal. User 4 is talking on the mobile communication terminal, and it is obvious that user 4 is interested in the AR image object 58a of the mobile communication terminal. Therefore, sound interference is avoided by moving the ambient environmental sound source 50b, which is not of user 4's interest, to post-movement point 53.

[0100] Fig. 15 is a fourth flowchart of the sound-confusion control program 66 corresponding to the fourth embodiment. In the fourth flowchart of the sound-confusion control program 66 in Fig. 15, the same steps as those in the first flowchart of the sound-confusion control program 66 shown in Fig. 8 are given the same numbers, and their explanations will be omitted.

[0101] The fourth flowchart of the sound congestion control program 66 in FIG. 15 differs from the first flowchart of the sound congestion control program 66 in FIG. 8 in that new steps S50 and S51 are added instead of S11, and steps S14 and S15 are deleted.

[0102] S50, S51: When the sound contamination control program 66 starts (S10), the information terminal 1 receives a notification that the user 4 is using the mobile information terminal (S50). The AR object of the mobile information terminal is generated and displayed, for example, by the CGR viewing program 65. As a result, the AR object placement deployment processing unit 26 obtains the placement position of the AR object of the mobile communication terminal (S51).

[0103] S12: The ambient sound source direction detection unit 20 detects the sound source direction of the ambient sound.

[0104] S13: The placement correction processing unit 27 compares the placement position of the AR object obtained in S51 with the sound source direction of the ambient environmental sound detected in S12 to determine the possibility of sound congestion. Since it is obvious that the image being viewed by the user is an image object of the mobile communication terminal, the placement correction processing unit 27 determines that the image the user is gazing at is an AR image object of the mobile communication terminal. If the placement correction processing unit 27 determines that there is a high possibility of sound congestion (S13: YES), it selects a head-related transfer function corresponding to the sound source direction after the movement of the ambient environmental sound source (S16) and performs a convolution operation (S17) to move the ambient environmental sound source in S16 and S17, thereby avoiding sound congestion.

[0105] As described above, according to the fourth embodiment, it is possible to provide an information terminal equipped with a sound-contamination control process that avoids sound contamination between the acoustic signal of CGR content and the surrounding environmental sound, even when the user wears the information terminal and uses a portable second information terminal, and a sound-contamination control method for the information terminal.

[0106] Although the embodiments of the present invention have been described above, it goes without saying that the configurations embodying the technology of the present invention are not limited to the above-described embodiments and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described components. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Note that the components and relative sizes shown in the drawings are simplified and idealized to clearly explain the present invention, and the actual implementation may result in more complex shapes. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired if different ones are used.

[0107] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0108] Furthermore, some or all of the functions of the invention may be implemented in hardware, for example, by designing them using an integrated circuit, a general-purpose processor, or an application-specific processor. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A microprocessor unit, CPU, or the like may also be implemented in software by interpreting and executing an operating program. The scope of software implementation is not limited, and hardware and software may be used in combination. Furthermore, some or all of the functions may be implemented by a server. The server may be, for example, a local server, a cloud server, an edge server, or an online service, as long as it can communicate with other components via communications to execute its functions. Information such as programs, tables, and files that implement each function may be stored in memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0109] This embodiment includes the following invention: (Supplementary Note 1) An information terminal comprising: a processor; a communication interface for communication connection to a sound pickup and reproduction device; and a display, wherein the processor displays virtual reality image objects on the display in order to place virtual reality objects including virtual reality image objects and virtual reality audio objects in real space, detects a sound source direction of an ambient environmental sound source relative to the information terminal based on ambient environmental sound picked up by the sound pickup and reproduction device, determines whether sound emitted from the virtual reality audio object and the ambient environmental sound will cause sound mixing based on the placement position of the virtual reality audio object and the sound source direction of the ambient environmental sound source, and if sound mixing is determined to occur, moves the placement position of the virtual reality audio object or the sound source direction of the ambient environmental sound, and performs control to output from the sound pickup and reproduction device a stereophonic sound signal corresponding to the placement position of the virtual reality audio object after the movement or a stereophonic sound signal corresponding to the sound source direction of the ambient environmental sound source after the movement. (Supplementary Note 2) A sound contamination control method for an information terminal, wherein the information terminal: displays virtual reality image objects on a display in order to place virtual reality objects including virtual reality image objects and virtual reality sound objects in real space; acquires ambient environmental sound picked up by an audio pickup and reproduction device; detects a sound source direction, which is the direction of the ambient environmental sound source relative to the information terminal, based on the ambient environmental sound; determines whether the sound emitted from the virtual reality sound object and the ambient environmental sound will cause sound contamination based on the placement position of the virtual reality sound object and the sound source direction of the ambient environmental sound source; and if it is determined that sound contamination will occur, moves the placement position of the virtual reality sound object or the sound source direction of the ambient environmental sound; and controls the audio pickup and reproduction device to output a stereophonic signal corresponding to the placement position of the virtual reality sound object after the movement or a stereophonic signal corresponding to the sound source direction of the ambient environmental sound source after the movement.

[0110] 1: Information terminal 1A: Head-mounted display 1B: Information processing device 2: Sound collection and reproduction device 2a: Sound collection and reproduction device 2b: Sound collection and reproduction device 3: Information processing unit 4: User 5: Real space image 6: Line of sight 6a: Line of sight 7: Processor 8: Memory 10: Front camera 11: Distance measurement sensor 12: Line of sight camera 12L: Left line of sight camera 12R: Right line of sight camera 13: Motion sensor 14: Camera image processing unit 15: Distance measurement processing unit 16: Line of sight detection processing unit 17: Motion detection processing unit 18: Near field communication interface 19: Ambient environment sound level detection unit 20: Ambient environment sound source direction detection unit 21: Ambient environment sound processing unit 22: Control unit 23: Network interface 24: AR object receiving unit 25: AR object data storage unit 26 : AR object placement deployment processing unit 27 : Placement correction processing unit 28 : Head related transfer function determination processing unit 29 : Head related transfer function data storage unit 30 : Convolution calculation processing unit 31 : Display 32 : Input operation unit 33 : Internal bus 34 : Near field communication interface 35 : Left speaker 36 : Right speaker 37 : Left microphone array 38 : Right microphone array 39 : Noise reduction unit 40 a : Temporal head holder 40 b : Front holder 50 : Surrounding environment sound source 50 a : Real object 50 b : Surrounding environment sound source 51 a : AR image object 51 b : AR sound object 52 a : AR image object 52 b : AR sound object 53 : Post-movement point 54 a : Real object 54 b : Surrounding environment sound source 55 a : AR image object 55b: AR sound object 56a: Mosaic processing 56b: Blur processing 57a: AR image object 57b: AR sound object 58a: AR image object 58b: AR sound object 60: Communication interface 60a: Wireless LAN interface 60b: Near field communication interface 61: CPU 62: RAM 63: Flash memory 64: Basic operation program 65: CGR viewing program66: Sound interference control program 67: AR object data 68: Head-related transfer function data 69: Data section 70: Sensor group 70a: Gyro sensor 70b: Acceleration sensor 70c: Geomagnetic sensor 71: Internal bus

Claims

An information terminal, a processor; a communication interface for communication connection to the sound pickup and playback device; a display; The processor: displaying virtual reality objects, including virtual reality image objects and virtual reality sound objects, on the display to place the virtual reality objects in real space; Detecting a sound source direction, which is a direction of the ambient environmental sound source relative to the information terminal, based on the ambient environmental sound picked up by the sound pickup and reproduction device; determining whether the sound emitted from the virtual reality sound object and the ambient environmental sound will be mixed based on the placement position of the virtual reality sound object and the sound source direction of the ambient environmental sound source; If it is determined that there is sound congestion, the placement position of the virtual reality audio object or the sound source direction of the ambient environmental sound is moved; and performing control to output, from the sound pickup and reproduction device, a stereophonic signal corresponding to the position of the virtual reality sound object after the movement or a stereophonic signal corresponding to the sound source direction of the ambient environmental sound source after the movement. Information terminal.

2. The information terminal according to claim 1, The processor: applying a head-related transfer coefficient corresponding to the placement position of the virtual reality sound object to perform a convolution operation to generate a stereophonic sound signal of the virtual reality object; applying a head-related transfer coefficient corresponding to the sound source direction of the ambient environmental sound sound source to perform a convolution operation to generate a stereophonic signal of the ambient environmental sound; When moving the placement position of the virtual reality sound object, a convolution operation is performed by applying a head-related transfer coefficient corresponding to the placement position of the virtual reality sound object after the movement, and a stereophonic sound signal of the virtual reality object corresponding to the placement position after the movement is generated; When the sound source direction of the ambient environmental sound source is moved, a convolution operation is performed by applying a head-related transfer coefficient corresponding to the sound source direction of the ambient environmental sound source after the movement, and a stereophonic signal of the ambient environmental sound corresponding to the sound source direction after the movement is generated. Information terminal.

2. The information terminal according to claim 1, further comprising a gaze detection sensor for detecting the gaze of a user of the information terminal; The processor recognizes an image that the user is gazing at on the display, and if the user is gazing at the virtual reality image object, moves the sound source direction of the ambient environmental sound sound source, and if the user is gazing at a real object existing in real space, moves the virtual reality sound object. Information terminal.

2. The information terminal according to claim 1, further comprising an input operation unit that accepts an operation by a user of the information terminal; The processor receives a selection operation of a gaze image from the user via the input operation unit, and if the gaze image is the virtual reality image object, moves the sound source direction of the ambient environmental sound sound source, and if the gaze image is a real object existing in real space, moves the virtual reality sound object. Information terminal.

2. The information terminal according to claim 1, When the processor determines that the sound emitted from the virtual reality sound object and the surrounding environmental sound are mixed together, the processor moves the display position of the virtual reality image object on the display, thereby moving the placement position of the virtual reality sound object. Information terminal.

6. The information terminal according to claim 5, further comprising a gaze detection sensor for detecting the gaze of a user of the information terminal; The processor: While the display position of the virtual reality image object on the display is being moved, it is confirmed whether the user's line of sight is following the virtual reality image object based on a sensor output from the line of sight detection sensor. Information terminal.

2. The information terminal according to claim 1, the virtual reality object is a second information terminal; The processor: when it is determined that the sound emitted from the virtual reality object of the second information terminal and the ambient environmental sound are mixed together, the sound source of the ambient environmental sound is moved. Information terminal. 　 A sound congestion control method for an information terminal, comprising: The information terminal, Displaying virtual reality objects, including virtual reality image objects and virtual reality sound objects, on a display to place the virtual reality objects in a real space; acquiring a surrounding environmental sound picked up by a sound pickup and reproduction device, and detecting a sound source direction, which is a direction of the surrounding environmental sound source relative to the information terminal, based on the surrounding environmental sound; determining whether the sound emitted from the virtual reality sound object and the ambient environmental sound will be mixed based on the placement position of the virtual reality sound object and the sound source direction of the ambient environmental sound source; If it is determined that there is sound congestion, the placement position of the virtual reality audio object or the sound source direction of the ambient environmental sound is moved; and performing control to output, from the sound pickup and reproduction device, a stereophonic signal corresponding to the position of the virtual reality sound object after the movement or a stereophonic signal corresponding to the sound source direction of the ambient environmental sound source after the movement. A method for controlling noise pollution in an information terminal.

Images