Multi-channel audio signal playing method, head-mounted device and storage medium

Abstract

The application discloses a multi-channel audio signal playing method, a head-mounted device and a storage medium, and the method comprises the following steps: determining a corresponding visual angle direction of the head-mounted device and the azimuth information between each loudspeaker device and the head-mounted device; determining the mapping relationship between each channel of the multi-channel audio to be played and the loudspeaker device according to the visual angle direction and the azimuth information; and controlling the loudspeaker device to cooperatively play the multi-channel audio according to the mapping relationship, so that the auditory direction of the audio signal played by each channel of the multi-channel audio matches the direction of each channel. The application aims to improve the auditory immersion of the head-mounted device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio signals, in particular to a multi-channel audio signal playing method, a head-mounted device and a storage medium. BACKGROUND

[0002] Virtual reality products present a 360-degree visual field scene to users, and can move the visual content according to the rotation direction of the user to give the user a sense of being there.

[0003] In related technologies, virtual reality products generally output audio through loudspeakers on the device or connected earphones. Due to factors such as the structure and volume of virtual reality products, the loudspeaker capacity and number are limited, and the virtual reality product can only output monaural or two-channel stereo sound effects, and cannot achieve a stereo spatial panorama sound effect, so that the virtual reality product cannot provide a real experience to the user.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a multi-channel audio signal playing method, a head-mounted device and a storage medium, aiming to improve the auditory immersion effect of the head-mounted device.

[0006] To achieve the above purpose, the present application provides a multi-channel audio signal playing method applied to a head-mounted device, wherein the head-mounted device is connected with a plurality of sound broadcasting devices arranged at different positions, and the method comprises the following steps:

[0007] determining the corresponding visual angle direction of the head-mounted device and the positional information between each sound broadcasting device relative to the head-mounted device;

[0008] determining the mapping relationship between each channel of the multi-channel audio to be played and the sound broadcasting device according to the visual angle direction and the positional information;

[0009] controlling the sound broadcasting devices to cooperatively play the multi-channel audio according to the mapping relationship, so that the auditory direction of the audio signal played by each channel of the multi-channel audio matches the direction of each channel.

[0010] Optionally, the step of determining the mapping relationship between each channel of the multi-channel audio to be played and the sound broadcasting device according to the visual angle direction and the positional information comprises:

[0011] determining the sound broadcasting device whose positional information is consistent with the visual angle direction as the target sound broadcasting device corresponding to the front channel;

[0012] The mapping relationship is determined based on the relative position of each broadcasting device relative to the target broadcasting device, and the relative direction between each channel and the channel directly in front.

[0013] Optionally, the step of determining the mapping relationship between each channel of the multi-channel audio to be played and the broadcasting device based on the viewing direction and the orientation information includes:

[0014] The relative angle between each of the broadcasting devices and the viewing direction is determined based on the orientation information;

[0015] The mapping relationship is determined based on the comparison between the relative angle and the angle of each channel's direction relative to the direction of the front channel.

[0016] Optionally, the step of determining the viewing angle direction corresponding to the head-mounted device and the azimuth information of each of the broadcasting devices relative to the head-mounted device includes:

[0017] The viewing direction and the orientation information are determined based on the ultra-wideband connection between each of the aforementioned broadcasting devices and the head-mounted device; or

[0018] The viewing direction is determined based on the ultra-wideband connection between each of the broadcasting devices and the head-mounted device, and the orientation information is determined based on the viewing direction and the relative position information between the broadcasting devices.

[0019] Optionally, the step of controlling the broadcasting equipment to collaboratively play the multi-channel audio according to the mapping relationship includes:

[0020] Determine the audio signal corresponding to each channel in the multi-channel audio;

[0021] According to the mapping relationship, each audio signal is sent to the broadcasting device corresponding to the channel where the audio signal is located, and the broadcasting device receives and plays the audio signal.

[0022] Optionally, before the step of determining the audio signal corresponding to each channel in the multi-channel audio, the method further includes:

[0023] Acquire the visual scene of the head-mounted device and the corresponding audio source signal;

[0024] Determine the sound source direction of each audio signal in the visual scene relative to the sound source direction of the head-mounted device;

[0025] The corresponding channel of the audio signal is determined based on the direction of the sound source.

[0026] Optionally, before the step of receiving and playing the audio signal, the step of sending each audio signal to the corresponding broadcasting device of the sound channel where the audio signal is located according to the mapping relationship further comprises:

[0027] When the audio signal corresponding to the sound source direction does not match the orientation information of the broadcasting device corresponding to the sound channel where the audio signal is located, re-performing the step of determining the orientation information between the plurality of broadcasting devices and the head-mounted device.

[0028] Optionally, the step of determining the orientation information between the plurality of broadcasting devices and the head-mounted device comprises:

[0029] Obtaining a bird's-eye view image of the head-mounted device and the plurality of broadcasting devices;

[0030] Identifying the front end of the head-mounted device and the relative position of each broadcasting device relative to the head-mounted device according to the bird's-eye view image;

[0031] According to the orientation information of the front end of the head-mounted device, the relative position is determined according to the orientation information.

[0032] In addition, to achieve the above-mentioned purpose, the application also provides a head-mounted device, which comprises a memory, a processor and a multi-channel audio signal playing program stored in the memory and executable on the processor, and the multi-channel audio signal playing program realizes the steps of the multi-channel audio signal playing method when executed by the processor.

[0033] In addition, to achieve the above-mentioned purpose, the application also provides a computer readable storage medium, which stores a multi-channel audio signal playing program, and the multi-channel audio signal playing program realizes the steps of the multi-channel audio signal playing method when executed by a processor.

[0034] The embodiment of the present application provides a multi-channel audio signal playing method, a head-mounted device and a storage medium, which first determines a corresponding visual angle direction of the head-mounted device and orientation information between each of the sound playing devices relative to the head-mounted device; determines a mapping relationship between each channel of the multi-channel audio to be played and the sound playing devices according to the visual angle direction and the orientation information; and controls the sound playing devices to cooperatively play the multi-channel audio according to the mapping relationship, so that the hearing direction of the audio signal of each channel of the multi-channel audio after playing matches the direction of each channel. The sound playing devices are arranged around the head-mounted device, and the visual angle direction of the head-mounted device and the orientation information of each sound playing device are used to match the sound playing devices for each channel of the multi-channel audio signal, so that when the multi-channel video is played, the hearing direction of the audio signal of each channel heard by the user matches the direction of each channel, so that when the head-mounted device is used, a surround viewing effect can be obtained, the hearing immersion is improved, and the authenticity of the head-mounted device is improved. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 Fig. 1 is a terminal structure schematic diagram of a hardware running environment related to the embodiment of the present application;

[0036] Figure 2 Fig. 2 is a flowchart of an embodiment of the multi-channel audio signal playing method of the present application;

[0037] Figure 3 Fig. 3 is a flowchart of another embodiment of the multi-channel audio signal playing method of the present application;

[0038] Figure 4 Fig. 4 is a layout schematic diagram of the head-mounted device and the plurality of sound playing devices related to the embodiment of the present application.

[0039] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0040] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0041] In the related art, the head-mounted virtual reality device generally outputs an audio signal through a loudspeaker on the device or a connected earphone, and the loudspeaker on the device and the earphone can only realize a single-channel or double-channel stereo sound effect, and cannot achieve a panorama sound effect, so that the hearing immersion of the user is not high.

[0042] In order to improve the hearing immersion of the head-mounted device, the embodiment of the present application provides a multi-channel audio signal playing method, a head-mounted device and a storage medium, wherein the main steps of the method include:

[0043] determine a corresponding visual angle direction of the head-mounted device and orientation information between each of the sound playing devices and the head-mounted device;

[0044] determine a mapping relationship between each channel of the multi-channel audio to be played and the sound playing devices according to the visual angle direction and the orientation information;

[0045] control the sound playing devices to cooperatively play the multi-channel audio according to the mapping relationship, so that the auditory direction of the audio signal of each channel of the multi-channel audio after being played matches the direction of each channel.

[0046] In this way, the visual angle direction of the head-mounted device and the orientation information of each sound playing device are used to match the sound playing devices for each channel of the multi-channel audio signal, so that when the multi-channel video is played, the auditory direction of the audio signal of each channel heard by the user matches the direction of each channel, thereby improving the sense of immersion of the auditory.

[0047] The content claimed in the claims of the present application will be described in detail below with reference to the accompanying drawings.

[0048] As shown in Figure 1 , the terminal structure shown in Figure 1 is a schematic diagram of the hardware running environment involved in the embodiment of the present application.

[0049] The terminal of the embodiment of the present application can be a head-mounted device.

[0050] As shown in Figure 1 , the terminal can include a processor 1001, such as a CPU, a memory 1003, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The memory 1003 can be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1003 can also be an optional storage device independent of the aforementioned processor 1001.

[0051] Those skilled in the art can understand that the terminal structure shown in Figure 1 does not constitute a limitation on the terminal, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0052] As shown in Figure 1 , the memory 1003, as a computer storage medium, can include an operating system and a multi-channel audio signal playing program.

[0053] In the terminal shown in Figure 1 , the processor 1001 can be used to call the multi-channel audio signal playing program stored in the memory 1003 and perform the following operations:

[0054] determining a view direction corresponding to the head-mounted device and orientation information between the head-mounted device and each of the sound emitting devices;

[0055] determining a mapping relationship between each channel of the multi-channel audio to be played and each of the sound emitting devices according to the view direction and the orientation information;

[0056] controlling the sound emitting devices to cooperatively play the multi-channel audio according to the mapping relationship, so that the hearing direction of the audio signal of each channel of the multi-channel audio after being played matches the direction of each channel.

[0057] Further, the processor 1001 can invoke the multi-channel audio signal playing program stored in the memory 1003, and further perform the following operations:

[0058] determining the sound emitting device corresponding to the front channel as a target sound emitting device according to the orientation information and the view direction;

[0059] determining the mapping relationship according to the relative position of each sound emitting device relative to the target sound emitting device and the relative direction between each channel and the front channel.

[0060] Further, the processor 1001 can invoke the multi-channel audio signal playing program stored in the memory 1003, and further perform the following operations:

[0061] determining the relative angle between each of the sound emitting devices and the view direction according to the orientation information;

[0062] determining the mapping relationship according to the comparison result of the relative angle and the angle of the direction of each channel relative to the direction of the front channel.

[0063] Further, the processor 1001 can invoke the multi-channel audio signal playing program stored in the memory 1003, and further perform the following operations:

[0064] determining the view direction and the orientation information based on the ultra-wideband connection between each of the sound emitting devices and the head-mounted device; or

[0065] determining the view direction based on the ultra-wideband connection between each of the sound emitting devices and the head-mounted device, and determining the orientation information according to the relative position information between the view direction and each of the sound emitting devices.

[0066] Further, the processor 1001 can invoke the multi-channel audio signal playing program stored in the memory 1003, and further perform the following operations:

[0067] determining audio signals corresponding to each channel in the multi-channel audio;

[0068] sending each audio signal to a sound playing device corresponding to the channel where the audio signal is located according to the mapping relationship, and playing the audio signal by the sound playing device.

[0069] Further, the processor 1001 can call the multi-channel audio signal playing program stored in the memory 1003, and further perform the following operations:

[0070] obtaining a visual scene of the head-mounted device and audio source signals corresponding to the visual scene;

[0071] determining the audio source direction of each audio signal in the visual scene relative to the head-mounted device;

[0072] determining the channel corresponding to the audio signal according to the audio source direction.

[0073] Further, the processor 1001 can call the multi-channel audio signal playing program stored in the memory 1003, and further perform the following operations:

[0074] re-performing the step of determining the orientation information between the plurality of sound playing devices and the head-mounted device when the audio source direction corresponding to the audio signal does not match the orientation information of the sound playing device corresponding to the channel where the audio signal is located.

[0075] Further, the processor 1001 can call the multi-channel audio signal playing program stored in the memory 1003, and further perform the following operations:

[0076] obtaining a bird's eye view image of the head-mounted device and the plurality of sound playing devices;

[0077] identifying the front end of the head-mounted device and the relative position of each sound playing device relative to the head-mounted device according to the bird's eye view image;

[0078] determining the orientation information according to the relative position and the visual angle direction of the front end of the head-mounted device.

[0079] The following will be explained and described by specific exemplary schemes to better understand the protection scope of the claims of the present application. It can be understood that the following exemplary schemes do not limit the protection scope of the present application, but only serve to explain the present application.

[0080] Exemplarily, with reference to Figure 2 In an embodiment of the multi-channel audio signal playing method of the present application, the multi-channel audio signal playing method comprises the following steps:

[0081] Step S10: Determine the viewing angle direction corresponding to the head-mounted device, and the azimuth information of each of the broadcasting devices relative to the head-mounted device;

[0082] In this embodiment, the device that performs multi-channel audio signal playback can be a head-mounted device, which can be a virtual reality device or an augmented reality device. Due to their nature, virtual reality devices and augmented reality devices need to provide users with sufficient realism. They not only require sufficient visual immersion but also sufficient auditory immersion. Simply placing speakers on the head-mounted device is insufficient to provide adequate immersion; therefore, the head-mounted device can be connected to multiple audio playback devices positioned at different locations. To provide users with auditory experiences from different directions, multiple audio playback devices can be arranged around the head-mounted device, and these devices are used to play multi-channel signals.

[0083] Optionally, the audio equipment can be evenly arranged around the head-mounted device: the head-mounted device can be connected to eight audio equipment, arranged around the head-mounted device at the front, rear, left, right, left front, left rear, right front, and right rear. The multi-channel audio to be played on the head-mounted device can include eight channels, namely: front channel, rear channel, left channel, right channel, left front channel, left rear channel, right front channel, and right rear channel. The audio signal of each channel can be played by the corresponding audio equipment at the front, rear, left, right, left front, left rear, right front, and right rear.

[0084] Alternatively, in many scenarios, installing multiple audio devices with a head-mounted device is inconvenient. For example, when using a head-mounted device at home, installing multiple audio devices takes up space, is unsightly, and is expensive. Therefore, devices with built-in speakers in the room can be used as audio devices. The head-mounted device can broadcast Bluetooth signals, and devices with built-in speakers in the room, such as dedicated audio devices for the head-mounted device, mobile phones, tablets, computers, smart speakers, robot vacuums, and other smart appliances, can respond to the Bluetooth signal and establish a connection with the head-mounted device, thus simplifying the required equipment.

[0085] When the audio to be played needs to be played, the head-mounted device is detected to have a large rotation, the device is powered on, the mapping relationship is inaccurate, and the like, the head-mounted device establishes a connection with each loudspeaker device, and the direction of the visual angle of the head-mounted device is determined. The direction of the visual angle of the head-mounted device refers to the direction of the front end of the head-mounted device and the face of the user after the user wears the head-mounted device. The front end of the head-mounted device is consistent with the direction of the face of the user. The orientation information of each loudspeaker device relative to the head-mounted device also needs to be determined. The loudspeaker device is arranged at different orientations of the head-mounted device, and is irrelevant to the direction of the visual angle. Instead, the orientation information refers to the direction of the loudspeaker device relative to the head-mounted device, and can be determined according to the relative positions between each loudspeaker device and the head-mounted device.

[0086] Optionally, the direction of the visual angle and the orientation information are determined based on the ultra-wideband connection between each loudspeaker device and the head-mounted device; or the direction of the visual angle is determined based on the ultra-wideband connection between each loudspeaker device and the head-mounted device, and the orientation information is determined according to the relative position information between the direction of the visual angle and the loudspeaker device.

[0087] The UWB (Ultra Wide Band) module can be arranged on the head-mounted device and at least one loudspeaker device. Based on the UWB modules on the head-mounted device and the loudspeaker device, the ultra-wideband connection between the head-mounted device and the at least one loudspeaker device can be established. Based on the ultra-wideband connection, the relative positions of the head-mounted device and the loudspeaker device and / or the deviation of the relative direction of the loudspeaker device and the head-mounted device from the direction of the visual angle can be determined. The head-mounted device can establish an ultra-wideband connection with each loudspeaker device, so as to determine the direction of the visual angle of the head-mounted device based on the ultra-wideband connection, and determine the orientation information according to the relative positions of each head-mounted device and the loudspeaker device. The ultra-wideband connection can also be established with one or a small number of loudspeaker devices, the deviation of the relative direction of the loudspeaker device and the head-mounted device from the direction of the visual angle is determined based on the ultra-wideband connection, and then the direction of the visual angle of the head-mounted device and the orientation information of the loudspeaker device of the ultra-wideband connection are determined. Then, the orientation information of each loudspeaker device is determined according to the pre-set or real-time confirmed relative position information between the loudspeaker devices.

[0088] Optionally, a bird's-eye view image of the head-mounted device and the plurality of loudspeaker devices is acquired; the front end of the head-mounted device and the relative positions of each loudspeaker device relative to the head-mounted device are identified according to the bird's-eye view image; the orientation information is determined according to the front end of the head-mounted device, the direction of the visual angle, and the relative positions.

[0089] Head-mounted displays and audio equipment need to be placed in the same space and within a certain distance to provide users with a better auditory experience. Therefore, a camera is installed above the space where the head-mounted displays and audio equipment are located. Positioned at the center of multiple audio equipment units, the camera can capture a clear overhead view, thus more accurately determining their location. Based on the overhead images of the head-mounted displays and audio equipment, the camera identifies the front end of the head-mounted displays and the relative positions of each audio equipment unit relative to the head-mounted displays. The direction the front end faces is used as the visual direction, and the relative positions of each audio equipment unit relative to the head-mounted displays determine their orientation. For better identification, illuminated markers can be placed on the front end of the head-mounted displays; the position of these markers determines the front end of the head-mounted displays.

[0090] Step S20: Determine the mapping relationship between each channel of the multi-channel audio to be played and the broadcasting device based on the viewing direction and the orientation information;

[0091] Step S30: Control the broadcasting device to play the multi-channel audio in coordination according to the mapping relationship, so that the auditory direction of the audio signal of each channel in the multi-channel audio is matched with the direction of each channel.

[0092] In this embodiment, after the head-mounted device determines the viewing angle and the azimuth information of each broadcasting device relative to the head-mounted device, the azimuth of each broadcasting device is mapped to each channel of the multi-channel audio to be played. Each broadcasting device has a direction relative to the head-mounted device, and the audio signal under each channel also has a direction in the operating environment of the head-mounted device; this can be called the sound source direction of the audio signal. Based on the direction of the audio signal in the operating environment of the head-mounted device, the corresponding channel for transmission is determined. Generally, one channel corresponds to one direction, and the direction of the channel is the sound source direction of the audio signal under that channel. A mapping relationship is obtained by matching the azimuth information of the broadcasting devices with the directions corresponding to the channels.

[0093] Each audio device needs to play multi-channel audio in coordination according to the mapping relationship so that the auditory direction of the audio signal under each channel is consistent with the direction of each channel. The visual direction refers to the user's face orientation, and the mapping relationship needs to be determined together based on the viewing angle and orientation information so that the sound source direction of the corresponding audio signal under each channel is consistent with the user's auditory direction.

[0094] Optionally, the broadcasting device whose orientation information is consistent with the viewing direction is determined as the target broadcasting device corresponding to the front channel; the mapping relationship is determined according to the relative position of each broadcasting device relative to the target broadcasting device, and the relative direction between each channel and the front channel.

[0095] The mapping relationship of a group of sound channels and sound playing devices can be matched in multiple sound channels and multiple sound playing devices, and then the matching is sequentially performed according to the direction of each sound channel and the relative position relationship between the sound playing devices. First, the sound playing device with the same orientation information as the visual angle direction of the head-mounted device is determined as the target sound playing device corresponding to the front sound channel, and the mapping relationship between the target sound playing device and the front sound channel is established. When the multi-channel audio signal to be played includes the front sound channel audio signal, the front sound channel audio signal is played through the target sound playing device, so that the user has the auditory illusion that the sound is emitted from the front. After the front sound channel and the corresponding sound playing device are determined, the corresponding relationship between the other sound playing devices and the other sound channels is determined according to the relative position relationship between the sound playing devices, for example, the first sound playing device in the counterclockwise direction of the target sound playing device is determined as the a device according to the relative position relationship, and the first sound channel in the counterclockwise direction of the front is determined as the left front sound channel, so that the a device and the left front sound channel are matched, and the mapping relationship therebetween is established.

[0096] Optionally, the relative angle between each sound playing device and the visual angle direction is determined according to the orientation information; and the mapping relationship is determined according to a comparison result of the relative angle and the angle of the direction of each sound channel relative to the direction of the front sound channel.

[0097] The relative angle between the sound playing device and the visual angle direction is determined according to the orientation information, and the relative angle is the included angle between the line connecting the sound playing device and the head-mounted device and the visual angle direction. The relative angle range between the visual angle direction and each sound channel corresponding to the visual angle direction is determined according to the direction of each sound channel. If the relative angle between the sound playing device and the visual angle direction is within the angle range, the sound channel corresponding to the angle range and the sound playing device are matched, and the mapping relationship therebetween is established.

[0098] For better understanding, refer to Figure 4 :

[0099] (1) According to the visual angle direction of the head-mounted device, the sound channels of each sound playing device are distributed and matched in position; for example, the visual angle direction is towards the sound playing device 1, the sound playing device 1 can be distributed to play the front sound channel data, the sound playing device 3 can be distributed to play the left sound channel data, and the sound playing device 7 can be distributed to play the right sound channel data. By analogy, the sound playing device 2 corresponds to the left front, the sound playing device 8 corresponds to the right front, the sound playing device 4 corresponds to the left rear, the sound playing device 6 plays the right rear sound channel data, and the sound playing device 5 plays the rear sound channel data.

[0100] (2) When the user turns from position a to position b, the visual angle direction changes, and the animal's call sound (emitted by the sound playing device 5) in the rear of the scene changes to the left rear to emit the call sound, which truly restores the scene of sound change.

[0101] (3) When the visual scene does not match the audio scene, re-correct the mapping relationship between the sound equipment and the sound channel.

[0102] In the technical solution disclosed in the embodiment, the orientation direction corresponding to the head-mounted device and the orientation information between each sound equipment relative to the head-mounted device are determined, the mapping relationship between each sound channel of the multi-channel audio to be played and the sound equipment is determined according to the orientation direction and the orientation information, and the sound equipment is controlled to cooperatively play the multi-channel audio according to the mapping relationship, so that the auditory direction of the audio signal of each sound channel after being played matches the direction of each sound channel. In this way, the mapping relationship between each sound equipment and sound channel that can make the auditory direction of the user match the direction of the sound channel is determined through the orientation direction of the head-mounted device and the orientation information of each sound equipment, the user can hear the sound coming from different directions when playing the multi-channel signal, which is the same as the auditory state of the real user, thereby improving the immersion of the user when wearing the head-mounted device and improving the user experience.

[0103] Optionally, with reference to any of the above embodiments, in another embodiment of the multi-channel audio signal playing method of the application, the multi-channel audio signal playing method further comprises: Figure 3

[0104] Step S31, determining the audio signal corresponding to each sound channel in the multi-channel audio;

[0105] In this embodiment, the multi-channel audio includes audio signals of multiple sound channels, and the audio signals under each sound channel are not necessarily sent at the same time, so when the sound equipment is controlled to cooperatively play the multi-channel audio, the multi-channel audio is analyzed to determine the corresponding audio signal under each sound channel.

[0106] Optionally, the visual scene of the head-mounted device and the audio source signal corresponding to the visual scene are acquired, the audio source direction of each audio signal in the visual scene relative to the head-mounted device is determined, and the sound channel corresponding to the audio signal is determined according to the audio source direction.

[0107] ​The head-mounted device can also be a head-mounted display device, such as a virtual reality device and an augmented reality device. Through virtual display technology, a visual scene is presented to the user, so that the user has a visual immersion when viewing the visual scene of the head-mounted device, and thus feels as if he is in the visual scene. The visual scene provides a 360-degree three-dimensional spatial experience for the user. In the visual scene, there is a sound source emitting an audio signal, for example, in a driving environment, the sound source is an oncoming car, and the motor sound moves from far to near and from front to back as the sound source moves. The audio signal (motor sound) corresponding to the visual scene is obtained. The sound source direction of the audio signal (car) in the visual scene relative to the head-mounted device is determined (the sound source direction changes from left front to left rear). The sound source direction is determined according to the sound source direction (from the left front channel to the left rear channel).

[0108] Step S32, according to the mapping relationship, each audio signal is sent to the corresponding broadcast device of each audio signal in the sound channel, and the broadcast device receives and plays the audio signal.

[0109] In this embodiment, the broadcast device establishes a communication connection, such as a Bluetooth connection, with the head-mounted device. After determining the corresponding audio signal under each sound channel, the corresponding broadcast device of each sound channel is determined according to the mapping relationship, and the corresponding audio signal under each sound channel is sent to the corresponding broadcast device based on the communication connection. After receiving the audio signal, the broadcast device plays the audio signal, thereby realizing the effect of playing the audio signal in different directions.

[0110] Optionally, when the audio signal corresponding to the sound source direction does not match the orientation information of the broadcast device corresponding to the sound channel where the audio signal is located, the step of determining the orientation information between the plurality of broadcast devices and the head-mounted device is re-executed.

[0111] In order to improve the matching of vision and hearing, the orientation information of the broadcast device playing the audio signal needs to be consistent with the orientation information of the broadcast device playing the audio signal. If the deviation between the orientation information of the broadcast device and the orientation information of the broadcast device is greater than a preset value, it is determined that the two do not match, and step S10 needs to be re-executed to correct the mapping relationship.

[0112] In the technical scheme disclosed in this embodiment, the audio signal corresponding to each sound channel in the multi-channel audio is determined; according to the mapping relationship, each audio signal is sent to the corresponding broadcast device of each audio signal in the sound channel, and the broadcast device receives and plays the audio signal, thereby realizing the cooperative playing of the plurality of broadcast devices for the multi-channel audio, and determining the direction of each audio signal in the multi-channel audio through the sound source direction of each audio signal, and playing through the broadcast device with the same orientation, improving the coupling of vision and hearing, improving the immersion of audio and video, and improving the reality of the scene of the head-mounted device.

[0113] In addition, the embodiment of the present application also provides a head-mounted device, which comprises a memory, a processor and a multi-channel audio signal playing program stored in the memory and executable on the processor, and the multi-channel audio signal playing program realizes the steps of the multi-channel audio signal playing method according to the above various embodiments when executed by the processor.

[0114] In addition, the embodiment of the present application also provides a computer readable storage medium, which stores a multi-channel audio signal playing program, and the multi-channel audio signal playing program realizes the steps of the multi-channel audio signal playing method according to the above various embodiments when executed by a processor.

[0115] It should be noted that, in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0116] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) as described above, and includes a plurality of instructions for enabling a head-mounted device to execute the method described in each embodiment of the present application.

[0118] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation according to the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-channel audio signal playback method, characterized by, The application is applied to a head-mounted device connected with multiple sound broadcasting devices arranged at different positions, and the method comprises: determining the corresponding visual direction of the head-mounted device and the positional information between each sound broadcasting device relative to the head-mounted device; wherein the visual direction refers to the orientation of the head-mounted device and the user's face after the user wears the head-mounted device, and the front end of the head-mounted device is consistent with the direction of the user's face; determining the mapping relationship between each channel of the multi-channel audio to be played and the sound broadcasting device according to the visual direction and the positional information; controlling the sound broadcasting devices to cooperatively play the multi-channel audio according to the mapping relationship, so that the auditory direction of the audio signal of each channel of the multi-channel audio after playing matches the direction of each channel; wherein the step of controlling the sound broadcasting devices to cooperatively play the multi-channel audio according to the mapping relationship comprises: obtaining the visual scene of the head-mounted device and the corresponding sound source signal; determining the sound source direction of each audio signal in the visual scene relative to the head-mounted device; determining the corresponding channel of the audio signal according to the sound source direction; determining the corresponding audio signal of each channel in the multi-channel audio; sending each audio signal to the sound broadcasting device corresponding to the channel where the audio signal is located according to the mapping relationship, and the sound broadcasting device receives and plays the audio signal; wherein the channels of each sound broadcasting device are allocated according to the visual orientation of the head-mounted device, wherein when the visual orientation is sound broadcasting device 1, sound broadcasting device 1 plays the front channel data, sound broadcasting device 3 plays the left channel data, sound broadcasting device 7 plays the right channel data, sound broadcasting device 2 plays the left front channel data, sound broadcasting device 8 plays the right front channel data, sound broadcasting device 4 plays the left rear channel data, sound broadcasting device 6 plays the right rear channel data, and sound broadcasting device 5 plays the rear channel data; when the user turns, the visual orientation changes, and when the position a changes to the position b, the visual orientation changes, and the animal call sound emitted by the rear sound broadcasting device 5 changes to the call sound emitted by the left rear sound broadcasting device 4; when the visual scene and the audio scene do not match, the mapping relationship between the sound broadcasting device and the channel is re-corrected.

2. The multi-channel audio signal playback method of claim 1, wherein, The step of determining the mapping relationship between each channel of the multi-channel audio to be played and the sound broadcasting device according to the visual direction and the positional information comprises: determining the sound broadcasting device corresponding to the front channel as the target sound broadcasting device according to the positional information and the visual direction; determining the mapping relationship according to the relative position of each sound broadcasting device relative to the target sound broadcasting device and the relative direction between each channel and the front channel.

3. The multi-channel audio signal playback method of claim 1, wherein, The step of determining the mapping relationship between each channel of the multi-channel audio to be played and the sound broadcasting device according to the visual direction and the positional information comprises: determining the relative angle between each sound broadcasting device and the visual direction according to the positional information; determining the mapping relationship according to the comparison result of the relative angle and the angle of the direction of each channel relative to the direction of the front channel.

4. The multi-channel audio signal playback method of claim 1, wherein, The step of determining the visual angle direction corresponding to the head-mounted device and the orientation information between each of the sound broadcasting devices and the head-mounted device comprises: determining the visual angle direction and the orientation information based on the ultra-wideband connection between each of the sound broadcasting devices and the head-mounted device; or determining the visual angle direction based on the ultra-wideband connection between each of the sound broadcasting devices and the head-mounted device, and determining the orientation information according to the relative position information between the visual angle direction and the sound broadcasting device.

5. The multi-channel audio signal playback method of claim 1, wherein, Before the step of receiving and playing the audio signal by the sound broadcasting device corresponding to the sound channel of the audio signal, the method further comprises: when the audio signal corresponding to the sound source direction does not match the orientation information of the sound broadcasting device corresponding to the sound channel of the audio signal, re-executing the step of determining the orientation information between the plurality of sound broadcasting devices and the head-mounted device.

6. The multi-channel audio signal playback method of claim 1, wherein, The step of determining the orientation information between the plurality of sound broadcasting devices and the head-mounted device comprises: obtaining a top-down image of the head-mounted device and the plurality of sound broadcasting devices; identifying the front end of the head-mounted device and the relative position of each of the sound broadcasting devices relative to the head-mounted device according to the top-down image; determining the orientation information according to the relative position according to the visual angle direction of the front end of the head-mounted device.

7. A head-mounted device, characterized by The head-mounted device comprises a memory, a processor, and a multi-channel audio signal playing program stored on the memory and executable on the processor, and the multi-channel audio signal playing program, when executed by the processor, implements the steps of the multi-channel audio signal playing method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a multi-channel audio signal playing program, and the multi-channel audio signal playing program, when executed by the processor, implements the steps of the multi-channel audio signal playing method according to any one of claims 1 to 6.

Images