Video data recording method, video data reproduction method, and stereophonic sound collection device

Abstract

Provided is a video data recording method comprising: a step in which a processor acquires an image signal which is generated after a camera captures an image, a stereophonic signal which is generated after sound is collected from a left microphone and a right microphone at the imaging timing of the camera, and a camera motion signal indicating motion of the camera at the imaging timing, and a microphone motion signal indicating motion of the left microphone and the right microphone at the imaging timing; a step in which the processor uses the camera motion signal and the microphone motion signal to calculate an azimuth angle deviation in the left-right center direction of the left microphone and the right microphone with respect to the center of the imaging direction of the camera; and a step in which the processor corrects the azimuth angle deviation with respect to the stereoscopic sound signal, and stores, in a memory, video data in which the image signal is associated with the corrected stereophonic signal.

Description

Video data recording method, video data playback method, and 3D sound pickup device

[0001] The present invention relates to a video data recording method, a video data playback method, and a stereophonic sound pickup device, and in particular to an information terminal that picks up and / or plays back (hereinafter referred to as sound pickup and playback) a stereophonic signal in synchronization with a video, and a sound signal control method for the information terminal.

[0002] Patent Document 1 discloses a listening system, stating, "The present invention provides a method for applying a listening system to a head to provide a listener with binaural recordings. The binaural recordings are listened to using a listening device, the binaural recordings comprising a left binaural ear signal for the listener's left ear and a right binaural ear signal for the listener's right ear. The method comprises the steps of determining a head orientation, determining a sound source direction of the binaural recording relative to the head orientation, detecting a change in head orientation to the new head orientation, and adapting the binaural recording to take into account the sound source direction of the binaural recording and the new head orientation." (Abstract excerpt)

[0003] Special Publication No. 2021-535648

[0004] However, in order to record stereoscopic video, when connecting a pair of earphones with built-in microphones (hereinafter referred to as an audio pickup and playback device) to an information terminal capable of capturing image signals such as stereoscopic image signals, thereby simultaneously recording image signals and stereoscopic audio signals, it is necessary to take into account not only the head movement of the person collecting the sound, but also changes in the camera shooting direction of the information terminal, but this is not mentioned in Patent Document 1.

[0005] The present invention has been made in consideration of the above points, and its purpose is to provide a video data recording method, a video data playback method, and a stereophonic sound recording device that more faithfully reproduce the positional relationship between the field of view of the sound collector when recording a stereoscopic image and the direction of the sound source heard by the sound collector.

[0006] In order to solve the above problems, the present invention has the configuration described in the claims. One example of such a method for recording video data includes the steps of: acquiring, by a processor, image signals generated by a camera capturing an image; stereophonic signals picked up by a left microphone and a right microphone at the timing of capturing an image with the camera; a camera motion signal indicating the movement of the camera at the capturing timing; and a microphone motion signal indicating the movement of the left microphone and the right microphone at the capturing timing; calculating, using the camera motion signal and the microphone motion signal, a deviation in azimuth angle of the left and right microphones in the center direction of the left and right microphones relative to the center of the camera's capturing direction; correcting the deviation in azimuth angle for the stereophonic signals; and storing, in a memory, video data in which the image signals are associated with the corrected stereophonic signals.

[0007] According to the present invention, it is possible to provide a video data recording method, a video data playback method, and a stereophonic sound pickup device. Objects, configurations, and effects other than those described above will be described in the following embodiments.

[0008] FIG. 1 is a block diagram of an information terminal according to a first embodiment. FIG. 1 is a diagram illustrating the operation of correcting the sound source direction when collecting a stereophonic signal with an information terminal according to the present invention. FIG. 2 is a diagram illustrating the operation of correcting the sound source direction when collecting a stereophonic signal with an information terminal according to the present invention. FIG. 3 is a diagram illustrating the operation of correcting the sound source direction when playing a stereophonic signal with an information terminal according to the present invention. FIG. 4 is a diagram illustrating the operation of correcting the sound source direction when playing a stereophonic signal with an information terminal according to the present invention. FIG. 5 is a block diagram of an information terminal according to a second embodiment. FIG. 6 is a diagram illustrating the operation of correcting the sound source direction only when playing a stereophonic signal with an information terminal according to the second embodiment. FIG. 7 is a diagram illustrating the operation of correcting the sound source direction only when playing a stereophonic signal with an information terminal according to the second embodiment. An example in which the information terminal according to the present invention is configured as an information processing device. FIG. 8 is a flowchart of the process of correcting the sound source direction during sound collection and playback with an information processing device. FIG. 9 is a flowchart of the process of correcting the sound source direction only during playback with an information processing device. 1 is an external view of an HMD (Head Mount Display) in an example where an information terminal is realized by the HMD; FIG. 2 is a block diagram of the HMD; FIG. 3 is an explanatory diagram of the operation of playing back a stereoscopic image signal and a stereoscopic sound signal superimposed on real space when viewing in MR (Mixed Reality) on the HMD; FIG. 4 is an explanatory diagram of the operation of playing back a stereoscopic image signal and a stereoscopic sound signal superimposed on real space when viewing in MR (Mixed Reality) on the HMD; and FIG. 5 is a flowchart of the process of correcting the direction of a sound source on the HMD.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings, components having the same functions are designated by the same reference numerals, and repeated description thereof will be omitted.

[0010] The stereophonic signal control technology of the present invention will contribute to the achievement of "9. Build resilient infrastructure, promote inclusive and sustainable industrialization, foster innovation and build resilient infrastructure" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0011] Before describing this embodiment in detail, the premise and problems of this embodiment will be described. Conventionally, there has been known an acoustic signal control method in which microphones are incorporated into a pair of left and right earphones, and a stereophonic signal is collected and stored in an information terminal by utilizing the difference between the acoustic signals collected by the pair of left and right microphones. The stored stereophonic signal can then be sent to the earphones, allowing the stereophonic signal to be heard. When collecting a stereophonic signal, the sound source direction of the stereophonic signal may change due to head movement of the person wearing the earphones. When the head position of the person collecting the sound differs from that of the listener, a phenomenon occurs in which the sound source direction of the stereophonic signal cannot be reproduced.

[0012] This phenomenon becomes more pronounced when capturing and playing back a stereoscopic video corresponding to a stereophonic signal. The term "stereophonic video" used here refers to a video that can be played back in synchronization with image signals generated by stereoscopic or two-dimensional photography and a stereophonic signal captured at the same time as the photography.

[0013] When recording a stereophonic video, the stereophonic video recording device includes an information terminal 1 (see FIG. 1) for shooting and an audio pickup / reproduction device 3 (see FIG. 1) worn on the head of the person collecting the stereophonic signal, and binaural recording is performed using the audio pickup / reproduction device 3. When one person holds the information terminal 1 and wears the audio pickup / reproduction device 3, the person shooting and the person collecting the sound are the same person. However, there are also cases where the person shooting using the information terminal 1 and the person collecting the sound wearing the audio pickup / reproduction device 3 are different people. In either case, since the information terminal 1 and the audio pickup / reproduction device 3 are configured separately, the movements of the information terminal 1 and the audio pickup / reproduction device 3 are not linked and can move independently. This can cause a discrepancy between the image perceived by the viewer when the stereophonic video is played and the sound image localization of the stereophonic signal when viewing it. This point will be explained further.

[0014] A viewer who plays back and watches stereoscopic video reproduces the image signal and visually recognizes the displayed image. For ease of explanation, it is assumed that the image is not captured by a panoramic camera, but rather is captured from an area with a viewing angle of, for example, 180 degrees. When the viewer looks toward the center of the displayed image, the viewer's line of sight coincides with the direction of the center of the image capture (or the capture range or field of view) of the information terminal 1 when the stereoscopic video was recorded.

[0015] Therefore, if the stereophonic signals are generated by listening to and collecting stereophonic signals with the left and right ears while the sound collector faces directly in the direction of the center of the image when recording a stereophonic video, the viewer will also face directly in the direction of the center of the image and listen to the stereophonic signals with the left and right ears to localize the sound image, so the sound image localization perceived by the sound collector while looking in the direction of the image will be nearly identical to the sound image localization perceived by the viewer while watching the image and playing back the stereophonic signals.

[0016] Therefore, when recording stereophonic video, it is necessary to align the shooting direction of the information terminal 1 with the front direction of the face of the person wearing the sound pickup and reproduction device 3 in order to reproduce the sound image positioning of the person picking up the sound to the viewer.

[0017] However, as described above, since the information terminal 1 and the sound pickup and reproduction device 3 move in independent directions, the direction in which the image is captured may not match the direction in which the face of the person picking up the sound when the stereophonic signal is picked up. If the stereophonic signal is reproduced while this mismatch remains, the viewer will experience a mismatch between the image and sound image localization that they are viewing.

[0018] The information terminal and the acoustic signal control method for the information terminal according to this embodiment are characterized in that the positional relationship between the position of the subject included in the image when the stereophonic audio video is recorded and the sound image position of the person recording the audio when viewing the image is reproduced for the viewer when the stereophonic audio video is played back.

[0019] [First Embodiment] A first embodiment will be described with reference to Figures 1 to 3A and 3B. Figure 1 is a block diagram of an information terminal 1 according to this embodiment, and also illustrates an acoustic pickup and playback device 3 for ease of explanation.

[0020] 1 includes a stereoscopic camera 10 that captures stereoscopic images, a motion sensor 11 (corresponding to a camera motion sensor), a processor 12, a memory 13, a communication interface (hereinafter referred to as "I / F") 14, an image display unit 15, an external input / output unit 16, and an input operation unit 17, all of which are interconnected by a bus 19. The functions of each unit will be described later.

[0021] When the processor 12 executes the 3D video recording and playback program, it realizes the functions of each of the following parts: a 3D image signal processing unit 121, a motion detection signal processing unit 122, a communication processing unit 123, a 3D sound signal and motion detection signal receiving unit 124, a correction angle calculation unit 125, a directional correction processing unit for 3D sound signals 126, a 3D image signal storage and playback processing unit 127, a 3D sound signal storage and playback processing unit 128, and a main control unit 129.

[0022] The memory 13 includes a head-related transfer function data section 131 consisting of an area in which head-related transfer function data is stored, a stereoscopic image signal data section 132 consisting of an area in which stereoscopic image signal data is stored, and a stereoscopic sound signal data section 133 consisting of an area in which stereoscopic sound signal data is stored.

[0023] The sound pickup and playback device 3 includes a proximity communication I / F 30, a motion sensor 31 (corresponding to a microphone motion sensor), a left speaker 32 (referred to as "Speaker L" in the figure), a left microphone 33 (referred to as "Microphone L" in the figure), a right speaker 34 (referred to as "Microphone R" in the figure), and a right microphone 35 (referred to as "Microphone R" in the figure).

[0024] The sound pickup and reproduction device 3 is a pair of devices, left and right, where the left side means that it is worn on the left ear side of the wearer, and the right side means that it is worn on the right ear side of the wearer.

[0025] The following describes the operation of collecting stereophonic signals using the information terminal 1 and the sound pickup and reproduction device 3. A person collecting the stereophonic signals wears the sound pickup and reproduction device 3 on each ear, holds the information terminal 1 in their hand, and captures an image of an object in front of them with the stereoscopic camera 10.

[0026] The stereoscopic camera 10 is, for example, a stereo camera, and the left camera obtains an image signal of the left eye's field of view, and the right camera obtains an image signal of the right eye's field of view.

[0027] The motion sensor 11 detects motions such as movement and rotation of the information terminal 1. The shooting direction of the stereoscopic camera changes according to the motion of the information terminal 1. The motion sensor 11 is configured using, for example, a gyro sensor, an acceleration sensor, and the like.

[0028] The stereoscopic image signal processing unit 121 combines the image signals from the left camera and the right camera that make up the stereoscopic imaging camera 10 into a pair of signals, and further sets and selects an image signal corresponding to the dominant eye of the sound collector or listener to create a two-dimensional image signal.

[0029] The stereoscopic image signal storage and playback processing unit 127 associates the stereoscopic image signal with the generation time and stores the signal in the stereoscopic image signal data unit 132 .

[0030] The motion detection signal processing unit 122 determines the orientation of the information terminal 1 from the motion detection signal of the motion sensor 11 .

[0031] The communication processing unit 123 supports multiple communication protocols, and includes, for example, network communication by connecting to an external network via the communication I / F 14 to share stereoscopic image signals, stereoscopic sound signals, etc., and proximity communication for sending and receiving stereoscopic sound signals and receiving motion detection signals with the sound pickup and playback device 3.

[0032] The main control unit 129 controls the operation settings and operation timing of the blocks connected to the bus 19 .

[0033] Furthermore, the image display unit 15 is configured with a flat display and displays the image signals captured by the stereoscopic camera 10, but the flat display displays the two-dimensional image signals selected by the stereoscopic image signal processing unit 121.

[0034] The external input / output unit 16 sends, for example, a stereoscopic image signal to an external 3D image display adapter, thereby enabling stereoscopic image display as an option.

[0035] The input operation unit 17 uses a touch sensor or the like to obtain user input operations.

[0036] The stereophonic signal and motion detection signal receiving unit 124 ("." means "and") receives the stereophonic signal and the motion detection signal from the sound pickup and reproduction device 3 via the communication processing unit 123, linking them to the time of generation, and sends the stereophonic signal to a stereophonic signal direction correction processing unit 126 and the motion detection signal to a correction angle calculation unit 125 via the bus 19.

[0037] 2A, 2B, and 2C. The correction angle calculation unit 125 receives the motion detection signal of the information terminal 1 from the motion detection signal processing unit 122 and calculates the correction angle described in Figures 2A, 2B, and 2C. The correction angle is used by the stereophonic signal direction correction processing unit 126, which corrects the sound source direction of the stereophonic signal by referring to the head-related transfer functions in the head-related transfer function data unit 131.

[0038] The stereophonic signal whose sound source direction has been corrected is stored in the stereophonic signal data unit 133 in association with the time of generation by the stereophonic signal storage and reproduction processing unit 128 .

[0039] Next, the operation of reproducing a stereophonic signal will be described.

[0040] The stereophonic signal storage and playback processing unit 128 reads out the stereophonic signal from the stereophonic signal data unit 133. The stereoscopic image signal storage and playback processing unit 127 reads out the selected 2D image signal from the stereoscopic image signal data unit 132 and displays it on the image display unit 15. The stereophonic image signal may be sent from the external input / output unit 16 to an external 3D image display adapter or the like, allowing stereophonic image display as an option.

[0041] The stereophonic signal / motion detection signal receiving unit 124 receives a motion detection signal from the sound pickup and reproduction device 3 worn by the listener and sends it to the correction angle calculation unit 125, which calculates the correction angle described in Figures 3A and 3B. The correction angle is used in the stereophonic signal direction correction processing unit 126, which refers to the head-related transfer functions in the head-related transfer function data unit 131 and corrects the sound source direction of the stereophonic signal read out from the stereophonic signal data unit 133.

[0042] The stereophonic signals whose sound source directions have been corrected are transmitted to the sound pickup and reproduction device 3 via the communication processing unit 123, and the listener can hear the signals from the speaker L32 and the speaker R34.

[0043] In the description of this embodiment, the sound pickup and playback device 3, which integrates a speaker and a microphone, is used to pick up and play back stereophonic signals, but it is also possible to use a sound pickup device with only a microphone for sound pickup and a sound playback device with only a speaker for playback.Furthermore, the information terminal 1 may perform sound pickup and playback using separate information terminals 1 that perform the respective functions.

[0044] Furthermore, in the explanation of this embodiment, an example has been given in which a stereoscopic image signal is generated and played back using the stereoscopic camera 10. However, the method of correcting the sound source direction of a stereoscopic sound signal explained in this embodiment can also be applied to cases in which two-dimensional image signals captured by a two-dimensional camera and a stereoscopic sound signal for correcting the sound source direction are saved and played back.

[0045] Furthermore, the microphone L33 and microphone R35 of the sound pickup and playback device 3 described in this embodiment are assumed to be microphones that are placed inside the wearer's ears to perform binaural sound pickup, but they may also be microphone arrays consisting of multiple microphones.

[0046] Furthermore, an in-camera for photographing the user's face may be provided, and a motion detection signal based on the movement of the user's eyes may be used instead of the motion sensor of the sound pickup and reproduction device 3 .

[0047] Furthermore, when reproducing image signals or stereophonic signals, a content management technique may be applied that checks the registration information of the user and the device being used before reproduction.

[0048] 2A and 2B are diagrams for explaining the operation of correcting the sound source direction when collecting a stereophonic signal with the information terminal of the present invention.

[0049] 2A shows a state in which a sound collector 2 faces an image object 41 present in the surrounding environment, and while pointing an information terminal 1 at the image object 41 to capture an image, the sound collector 2 wears a pair of left and right sound collection / reproduction devices 3L and 3R (hereinafter, both collectively referred to as 3) on the left and right ears of the sound collector 2 to perform binaural recording. Line L10 indicates the direction of the center of capture of the information terminal 1 at the start of capturing the stereophonic video, and line L30 indicates the center L30 of the sound collection / reproduction device 3 at the start of capturing the stereophonic video. It is assumed that a sound source 40 present in the surrounding environment is located to the left front of the sound collector 2.

[0050] The angle formed by the imaging center direction L10 of the information terminal 1 and the center L30 of the sound pickup and reproduction device 3 is 0, and the sound collector 2 starts collecting sound after confirming the image object 41 captured by the stereoscopic camera 10 of the information terminal 1 displayed on the flat display constituting the image display unit 15 of the information terminal 1. The angle between the imaging direction center L1 of the information terminal 1 and the sound source 40 is θ, and when the right side of the information terminal 1 is given a + symbol and the left side is given a - symbol, the sound source 40 is on the left side of the information terminal 1 in Figure 2A and is indicated as -θ.

[0051] Figure 2B shows a case where information terminal 1 has moved to δa and sound collector 2 has moved his / her head to angle δa from the state in Figure 2A. Sound pickup and reproduction device 3 worn by sound collector 2 picks up sound at an angle of -(θ+δa) with respect to sound source 40. Meanwhile, the angle between information terminal 1 and sound source 40 is -(θ+δa), and the angle between shooting center direction L11 and center L31 of sound pickup and reproduction device 3 is 0. Therefore, when looking at an angle of -(θ+δa) from sound source 40, the correct angle at which sound source 40 is heard is -(θ+δa), so correction of the stereophonic signal is not necessary in the state in Figure 2B.

[0052] On the other hand, Figure 2C shows the case where the information terminal 1 has moved δa and the sound collector 2 has moved his / her head φa from the state shown in Figure 2A. The sound collecting and reproducing device 3 worn by the sound collector 2 collects sound at an angle of -(θ+φa) with respect to the sound source 40. Meanwhile, the angle formed by the information terminal 1 and the sound source 40 is -(θ+δa), and the correct angle at which sound source 40 is heard is -(θ+δa). Therefore, the angle to be corrected is -(δa-φa). The sign - indicates that correction is required in the direction in which sound source 40 moves away from the information terminal 1 to the left.

[0053] 3A and 3B are diagrams illustrating the operation of correcting the sound source direction when a stereophonic signal is reproduced by the information terminal of the present invention. In Figures 3A and 3B, the same components as those in Figures 2A and 2B are assigned the same numbers. Figures 3A and 3B show the positional relationship between the sound source 42 of the reproduced stereophonic signal (corresponding to the sound source 40 when picked up) and the image 43 (corresponding to the image object 41 when photographed) displayed on the display of the information terminal 1.

[0054] In Fig. 2B, it was explained that correction of the stereophonic signal is not necessary, and Fig. 3A shows the state in which the audio recorded in Fig. 2B is played back. The stereophonic signal collected and saved in Fig. 3A indicates that the angle of the sound source direction heard without correction is -(θ+δa).

[0055] Here, when the viewer 5 moves by φb during playback of the stereophonic signal as shown in FIG. 3B, the angle at which the sound should be heard is −(θ+δa)−φb, and the correction angle to return to the state of FIG. 3A is −φb.

[0056] As described above, according to the first embodiment, when collecting stereophonic signals, if the direction of the sound pickup and reproduction device 3 deviates from the center of the image pickup while the information terminal 1 is facing the center of the image pickup, the deviation can be corrected before the stereophonic signal is reproduced. This allows the stereophonic video to be viewed by correctly reproducing the positional relationship between the image object captured in the image in the stereophonic video and the position of the sound source of the stereophonic signal when the image is being viewed.

[0057] Second Embodiment A second embodiment will be described with reference to FIG. 4, FIG. 5A and FIG. 5B.

[0058] Figure 4 shows an information terminal 1a of the second embodiment, which has a motion detection signal storage and playback processing unit 130 and a motion detection signal data unit 134 added to the information terminal 1 of the first embodiment described in Figure 1.

[0059] The information terminal 1a of the second embodiment does not perform angle correction when collecting sound, but associates the motion detection signal of the sound collection and reproduction device 3 with the collected stereophonic signal and the time of occurrence, and stores the signal in the motion detection signal data section 134 in the motion detection signal storage and reproduction processing section 130.

[0060] The motion detection signal storage / playback processing unit 130 also stores the motion detection signal of the information terminal 1 a in the motion detection signal data unit 134 .

[0061] During playback, in addition to the motion detection signal of the sound pickup and playback device 3 during playback, the motion detection signal of the sound pickup and playback device 3 during sound pickup and the motion detection signal of the information terminal 1a during sound pickup are supplied to the correction angle calculation unit 125, and the direction angle is corrected by the direction correction processing unit 126 for the stereophonic signal.

[0062] 5A and 5B are diagrams illustrating a correction operation for correcting the sound source direction only when a stereophonic signal is being played back. In FIGS. 5A and 5B, the same numbers are used to denote the same elements as in FIGS. 2A to 3B.

[0063] In Fig. 5A, the angle formed between the sound source direction of the stereophonic signal when it is picked up, i.e., when it is played back, and the sound pickup and reproduction device 3 is -(θ+φa), but when an angle deviation φb occurs between the information terminal 1a and the sound pickup and reproduction device 3 during playback as shown in Fig. 5B, taking into account the movement δa of the information terminal 1a when it picked up, the direction of the sound source of the stereophonic signal to be heard is -(θ+δa)+φb, as shown in Fig. 2C, and the correction angle is -(δa-φa)-φb. This is the sum of the correction angle during sound pickup and the correction angle during playback in the first embodiment.

[0064] As described above, the second embodiment has the same effect as the first embodiment, and in addition, compared to the case where the convolution calculation is performed twice for recording and playback by referring to the head related transfer function, it only needs to be performed once for playback, thereby reducing the processing load.

[0065] Third Embodiment A third embodiment will be described with reference to FIGS.

[0066] 6 shows an example in which the information terminal 1 of the first embodiment is configured as an information processing device 6, but for ease of explanation, a sound pickup and reproduction device 3 is also shown. The sound pickup and reproduction device 3 is the same as the device shown in FIG.

[0067] The information processing device 6 in Figure 6 includes a communication interface 60, a stereoscopic camera 61, a group of sensors 62, an image display unit 63, an external input / output interface 64, an input operation unit 65, a CPU 66, a RAM 67, and a flash memory 68, which are connected to each other by a bus 77.

[0068] The flash memory 68 has a basic operation program 69 such as an operating system (in the figure, the program is written as "PRG"), a stereoscopic image signal processing program 70, a stereophonic signal collection and playback program 71, and a data section 72 that stores a stereoscopic image signal 73, a stereophonic signal 74, a motion detection signal 75, and head-related transfer function data 76.

[0069] The communication interface 60 supports multiple communication protocols such as wireless LAN protocols and near field communication, etc. The wireless LAN is connected to an external network to share content such as stereoscopic image signals and stereoscopic sound signals with an external server, etc. In addition, the external server may assist the execution of the information processing device 6 by communicating the processing executed by the information terminal 8 with the external server.

[0070] The near field communication involves transmitting and receiving a stereophonic signal and a motion detection signal to and from the sound pickup and playback device 3 .

[0071] One of the roles of the sensor group 62 is as a motion sensor that includes a gyro sensor, an acceleration sensor, and the like, and detects the motion of the information processing device 6 .

[0072] The CPU 66 and RAM 67 constitute a computer system, and a basic operation program 69 , a stereoscopic image signal processing program 70 , and a stereophonic signal pickup and playback program 71 are loaded into the RAM 67 and executed by the CPU 66 .

[0073] The data section 72 may also store processing results that occur when the CPU 66 executes a program.

[0074] 7 is a flowchart of a stereophonic signal collection and reproduction program 71, which is a flowchart of the process of correcting the sound source direction during sound collection and reproduction. The stereophonic signal collection and reproduction program 71 runs in parallel with the stereophonic image signal processing program 70, and links the stereophonic image signal and the stereophonic sound signal by their generation times, and stores or reads them as a stereophonic image signal 73 and a stereophonic sound signal 74 in the data unit 72.

[0075] The program starts at step S10, and the operation mode is selected to be either sound collection or sound reproduction at step S11.

[0076] In the case of sound collection (S11: sound collection), in S12, the center of the shooting direction of the information terminal 1 is aligned with the direction of the sound collector's face (corresponding to the left-right center direction of the sound collection and reproduction device 3 worn by the sound collector), and the shooting reference direction (corresponding to L10 in FIG. 2A) and the sound collection reference direction (corresponding to L30 in FIG. 2A) are initially set. Thereafter, the information terminal 1 detects a movement angle L11 relative to L10 as a movement signal of the information terminal 1. The sound collection and reproduction device 3 detects a movement angle L31 relative to L30 as a movement signal of the sound collection and reproduction device 3. The stereophonic signal and the movement detection signal are received from the sound collection and reproduction device 3, and a movement detection signal is obtained from the information processing device 6 in S13.

[0077] Next, as explained in FIGS. 2A, 2B, and 2C, a correction angle is determined in S14, and the azimuth angle of the stereophonic signal is corrected in S15.

[0078] Furthermore, in S16, the stereophonic signal with the azimuth angle corrected is stored as the stereophonic signal 74 in the data section 72.

[0079] At S22, the termination condition is checked, and if it is to be terminated, the process ends at S23, and if it is not to be terminated, the process returns to before S11.

[0080] In the case of playback (S11: playback), the stereophonic signal 74 is read from the data section 72 in S17, and a motion detection signal of the sound pickup and playback device 3 is detected in S18.

[0081] Next, as explained in FIGS. 3A and 3B, a correction angle is determined in S19, and the azimuth angle of the stereophonic signal is corrected in S20.

[0082] Furthermore, in S21, the stereophonic signal with the azimuth angle corrected is transmitted to the sound pickup and reproduction device 3.

[0083] At S22, the termination condition is checked, and if it is to be terminated, the process ends at S23, and if it is not to be terminated, the process returns to before S11.

[0084] Fig. 8 is a flowchart of the stereophonic signal collection and reproduction program 71, which is a flowchart of the process of correcting the sound source direction only during reproduction. In the flowchart of Fig. 8, the same steps as in the flowchart of Fig. 7 are assigned the same numbers. The stereophonic signal collection and reproduction program 71 runs in parallel with the stereophonic image signal processing program 70, and links the stereophonic image signal and the stereophonic sound signal by their generation times, and stores or reads them out as a stereophonic image signal 73 and a stereophonic sound signal 74 in the data unit 72.

[0085] The program starts at step S10, and the operation mode is selected to be either sound collection or sound reproduction at step S11.

[0086] In the case of sound collection (S11: sound collection), a stereophonic signal and a motion detection signal are received from the sound collection and reproduction device 3 in S12, and a motion detection signal is obtained from the information processing device 6 in S13. The stereophonic signal and the two motion detection signals are stored in the data unit 72 in S30.

[0087] At S22, the termination condition is checked, and if it is to be terminated, the process ends at S23, and if it is not to be terminated, the process returns to before S11.

[0088] In the case of playback (S11: playback), the stereophonic signal and two motion detection signals are read from the data section 72 in S31, and the motion detection signal is received from the sound pickup and playback device 3 in S18. Then, as described with reference to Figures 5A and 5B, a correction angle is determined in S32, and the azimuth angle of the stereophonic signal is corrected in S20. Furthermore, the stereophonic signal with the corrected azimuth angle is transmitted to the sound pickup and playback device 3 in S21.

[0089] At S22, the termination condition is checked, and if it is to be terminated, the process ends at S23, and if it is not to be terminated, the process returns to before S11.

[0090] As described above, the third embodiment has the same effects as the first and second embodiments, and is characterized in that it can share many components with general-purpose information terminals, making it easy to realize an information terminal that collects and plays back stereophonic signals at low cost. Furthermore, it becomes possible to realize an acoustic signal control method for an information terminal that collects and plays back stereophonic signals by using a program.

[0091] [Fourth embodiment] A fourth embodiment will be described with reference to Figures 9 to 12. An information terminal 8 in the fourth embodiment is an example configured with AR (Argument Reality) glasses or an HMD (Head Mounted Display), and Figure 9 shows an external view of the HMD. As in Figure 1, sound pickup and reproduction devices 3aL and 3aR (sound pickup and reproduction device 3a divided into left and right halves) are also shown.

[0092] 9, the information terminal 8 includes a stereoscopic or two-dimensional imaging front camera 80, a distance measurement sensor 81, a sensor group 82, a stereoscopic image display unit 83, an input operation unit 84, an information processing unit 85, and holders 86a and 86b. A user of the HMD wears the HMD on their head using the holders 86a and 86b.

[0093] The image display unit 83 is a three-dimensional image display unit and is a semi-transmissive display or a non-transmissive display. In the case of MR, with a semi-transmissive display, the user visually recognizes the real space in front of them through a semi-transmissive screen. With a non-transmissive display, the user can confirm the real space in front of them by displaying a camera image capturing the real space in front of them on the non-transmissive display.

[0094] The front camera 80 is attached so as to capture an image of the real space in front of the user's line of sight, and the distance measurement sensor 81 measures the distance to a real object in the real space.

[0095] The information processing unit 85 receives and processes camera images captured by the front camera 80 and distance data measured by the distance sensor 81. It also has a built-in near-field communication interface, and transmits and receives stereophonic signals to and from the sound pickup and reproduction device 3a (shown separately as a right-side sound pickup and reproduction device 3aR and a left-side sound pickup and reproduction device 3aL in the figure), and also receives motion detection signals.

[0096] 10 is an example illustrating the configuration of the information terminal 8 shown in FIG. 9, particularly the information processing unit 85, which is configured as an information processing device similar to FIGS. 4 and 6. For ease of explanation, a sound pickup and reproduction device 3a is also illustrated. The sound pickup and reproduction device 3a may be the sound pickup and reproduction device 3 shown in FIG. 1 etc. without the motion sensor 31.

[0097] In FIG. 10, the same blocks as those in the HMD 8 in FIG. 9 are assigned the same numbers, and overlapping explanations will be omitted.

[0098] The information processing unit 85 includes a communication interface 60, an external input / output interface 64, a CPU 66, a RAM 67, and a flash memory 68, and blocks having the same functions as those in the information processing device 6 of FIG. 6 are given the same numbers.

[0099] The flash memory 68 has a data section 72 that stores basic operation programs 69 such as an operating system, a stereoscopic image signal processing program 70, a stereoscopic sound signal collection and playback program 71, a stereoscopic image signal 73, a stereoscopic sound signal 74, a motion detection signal 75, and head-related transfer function data 76.

[0100] The HMD 8 further includes an MR viewing program 87 for viewing MR. As will be described with reference to Fig. 11, the MR viewing program 87 forms an MR space in which a stereoscopic image signal captured by a camera and a stereoscopic sound signal collected by sound are superimposed on a real space and displayed as AR objects, and provides the MR space to the user of the HMD 8.

[0101] 11A and 11B illustrate the operation of displaying and reproducing a stereoscopic image signal and a stereoscopic sound signal in real space in a multiplexed manner during MR viewing on the HMD 8. In Figs. 11A and 11B, the same components as those in Figs. 3A and 3B are assigned the same numbers.

[0102] In FIG. 11A, a stereoscopic image signal 73 and a stereoscopic sound signal 74 are read from the HMD 8, and the MR viewing program 87 handles them as AR objects.

[0103] The stereoscopic image signal 73 is placed in the coordinate space of the real space as the AR object 45, and the stereoscopic sound signal 74 is placed in the coordinate space of the real space as the AR object 44. Note that a portion of the stereoscopic image signal 73 may be cut out and placed as the AR object 45. A real object 46 exists in the real space, and the viewer 5 wears an HMD 8 and views the real object 46 and the AR objects 44 and 45. The angle formed by the AR object 44 and the AR object 45 is the angle at the time of sound collection −(θ+δa).

[0104] 11B shows a state in which the viewer 5 has moved by an angle δc. Because the AR objects 44 and 45 are fixed in the real space as an MR space, the positional relationship with the real object 46 does not change with the movement of the viewer 5. Therefore, the angle of the direction in which the AR object should be heard is −(θ+δa)−δc, and the correction angle is −δc.

[0105] Figure 12 is a flowchart of the process for correcting the direction of a sound source in the HMD 8. The same steps as in the flowchart shown in Figure 7 are given the same numbers. This example makes it possible to correct the misalignment of the stereophonic signal with respect to the video in an information terminal that handles video and stereophonic signals.

[0106] Processing begins at S10. The stereophonic signal 74 is read from the data unit 72 at S17, and a motion detection signal from the HMD 8 is detected at S13. Next, a correction angle is determined at S40, and the azimuth angle of the stereophonic signal is corrected at S20. Then, the azimuth-corrected stereophonic signal is transmitted to the sound pickup and reproduction device 3a at S21.

[0107] At S22, the termination condition is checked, and if it is to be terminated, the process ends at S23, and if it is not to be terminated, the process returns to before S11.

[0108] It should be noted that the HMD 8 can also collect stereophonic signals. When an information terminal is worn on the user's head, as with the HMD 8, the movement of the information terminal coincides with the movement of the sound collecting and reproducing device. Therefore, the stereophonic signals can be stored without angle correction in the state where they are collected at -(θ+δa).

[0109] As described above, according to the fourth embodiment, even when an HMD or AR glasses is used as an information terminal, it is possible to collect and reproduce stereophonic signals.

[0110] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0111] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0112] Some or all of the functions of the present invention described above may be implemented in hardware, for example, by designing them using an integrated circuit, a general-purpose processor, or an application-specific processor. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. Furthermore, a microprocessor unit, a CPU, or the like may be implemented in software by interpreting and executing an operating program that implements each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Furthermore, some or all of the functions may be implemented by a server. Note that the server may be any type of server, such as a local server, cloud server, edge server, or internet service, as long as it can communicate with other components via communications to execute the functions. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0113] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0114] The above embodiments include the following inventions: (Supplementary Note 1) A video data recording method, comprising the steps of: a processor acquiring image signals captured by a camera and generated by the camera, stereophonic signals picked up by the left and right microphones at the timing of capturing images by the camera, and camera motion signals indicating the movement of the camera at the capturing timing and microphone motion signals indicating the movement of the left and right microphones at the capturing timing, calculating, using the camera motion signals and microphone motion signals, a deviation in azimuth angle in the center direction of the left and right microphones relative to the center of the camera's capturing direction, and correcting the deviation in azimuth angle for the stereophonic signals, and storing in a memory video data in which the image signals are associated with the corrected stereophonic signals. (Supplementary Note 2) A video data playback method, comprising the steps of: a processor reading video data to which correction angle data has been added that corrects for a deviation in azimuth angle in the center direction of the left and right of the left and right microphones used to pick up the stereophonic sound relative to the center of the shooting direction at the timing of shooting, outputting audio from the left and right speakers in which the deviation in azimuth angle has been corrected for the stereophonic signal using the correction angle data, and displaying a display image based on the image signal on a display. (Supplementary Note 3) A stereophonic sound pickup device comprising: a camera; a processor; a camera motion sensor that detects motion of the camera; a left microphone and a right microphone configured separately from the camera; a microphone motion sensor that detects the motion of each of the left microphone and the right microphone; and a memory, wherein the processor acquires stereophonic signals from the left microphone and the right microphone, calculates an azimuth angle deviation between the direction of the camera's shooting center and the direction of the centers of the left microphone and the right microphone based on the motion detection signal of the camera motion sensor and the motion detection signal of the microphone motion sensor while the left microphone and the right microphone are collecting sound, and corrects the azimuth angle deviation for the stereophonic signals and stores the corrected signals in the memory.

[0115] 1: Information terminal 1a: Information terminal 2: Sound collector 3: Sound pickup and reproduction device 3L: Left sound pickup and reproduction device 3R: Right sound pickup and reproduction device 3a: Sound pickup and reproduction device 3aL: Left sound pickup and reproduction device 3aR: Right sound pickup and reproduction device 5: User 6: Information processing device 8: Information terminal (HMD) 10: Stereoscopic camera 11: Motion sensor 12: Processor 13: Memory 14: Communication I / F 15: Image display unit 16: External input / output unit 17: Input operation unit 19: Bus 30: Near field communication I / F 31: Motion sensor 32: Left speaker 33: Left microphone 34: Right speaker 35: Right microphone 40: Sound source 41: Image object 42: Sound source 43 : Image 44 : AR object 45 : AR object 46 : Real object 60 : Communication interface 61 : Stereoscopic camera 62 : Sensor group 63 : Image display unit 64 : External input / output interface 65 : Input operation unit 66 : CPU 67 : RAM 68 : Flash memory 69 : Basic operation program 70 : Stereoscopic image signal processing program 71 : Stereophonic signal collection and reproduction program 72 : Data unit 73 : Stereoscopic image signal 74 : Stereophonic signal 75 : Motion detection signal 76 : Head-related transfer function data 77 : Bus 80 : Front camera 81 : Distance measurement sensor 82 : Sensor group 83 : Stereoscopic image display unit 84 : Input operation unit 85 : Information processing unit 86 a : Storage unit 86 b : Storage unit 87 : MR viewing program 121 : 3D image signal processing unit 122 : Motion detection signal processing unit 123 : Communication processing unit 124 : Motion detection signal receiving unit 125 : Correction angle calculation unit 126 : Direction correction processing unit 127 : 3D image signal storage and playback processing unit 128 : 3D sound signal storage and playback processing unit 129 : Main control unit 130 : Motion detection signal storage and playback processing unit 131 : Head-related transfer function data unit 132 : 3D image signal data unit 133 : 3D sound signal data unit 134 : Motion detection signal data unit

Claims

1. A video data recording method comprising the steps of: a processor acquiring image signals captured by a camera and generated by the camera, stereophonic signals picked up by the left and right microphones at the timing of the camera's capture, a camera motion signal indicating the movement of the camera at the timing of the capture, and a microphone motion signal indicating the movement of the left and right microphones at the timing of the capture; calculating, using the camera motion signal and the microphone motion signal, a deviation in azimuth angle in the center direction of the left and right microphones relative to the center of the camera's capture direction; and correcting the deviation in azimuth angle for the stereophonic signals, and storing in memory video data in which the image signals are associated with the corrected stereophonic signals.

2. A video data recording method according to claim 1, wherein the storing step generates correction angle data for correcting the deviation in the azimuth angle, corrects the deviation in the azimuth angle for the stereophonic signal using the generated correction angle data, and stores the video data and the correction angle data in the memory.

3. A video data playback method comprising the steps of: a processor reading the video data described in claim 2; outputting audio from the left and right speakers in which the deviation in azimuth angle has been corrected for the stereophonic signal using the correction angle data; and displaying a display image based on the image signal on a display.

4. A stereophonic sound pickup device comprising: a camera; a processor; a camera motion sensor that detects the movement of the camera; left and right microphones configured separately from the camera; a microphone motion sensor that detects the movement of each of the left and right microphones; and a memory, wherein the processor acquires stereophonic signals from the left and right microphones, calculates the azimuth angle deviation between the direction of the camera's center of photography and the central direction of the left and right microphones based on the motion detection signals of the camera motion sensor and the motion detection signals of the microphone motion sensor while the left and right microphones are collecting sound, and corrects the azimuth angle deviation for the stereophonic signals and stores the corrected signals in the memory.

5. A stereophonic sound pickup device according to claim 4, wherein the processor generates correction angle data for correcting the deviation of the azimuth angle, and stores the generated correction angle data and the stereophonic signal in association with each other in the memory.

6. A stereophonic sound pickup device according to claim 5, further comprising a speaker, wherein the processor corrects the deviation of the azimuth angle for the stereophonic signal based on the correction angle data, and outputs sound based on the corrected stereophonic signal from the speaker.

Images