Terminal and method for outputting multi-channel audio using multiple audio devices

Abstract

A terminal for outputting multi-channel audio by using multiple audio devices. The terminal may include: a camera; a communication interface configured to communicate with a plurality of first audio devices; and a processor configured to receive device information about the plurality of first audio devices through the communication interface or the camera; configure a multi-channel audio system including at least two second audio devices selected from among the plurality of first audio devices based on the device information; and output audio data through the second audio devices based on audio system information corresponding to the multi-channel audio system.

Description

Technical Field

[0001] The present disclosure relates to a terminal and method for outputting multi-channel audio using a plurality of audio devices. Background Art

[0002] Existing multi-channel audio systems include audio devices whose output channels are predetermined and pre-set, and professional assistance is required to arrange the audio devices for proper multi-channel audio output. Therefore, even if a user has multiple audio devices that form a multi-channel audio system, installing them in the appropriate locations can be challenging. Furthermore, configuring a multi-channel audio system requires equipment that combines various audio devices capable of outputting predetermined channels.

[0003] If a multi-channel audio system can be constructed by combining different mono audio devices or stereo audio devices installed in the home, it is possible to provide users with highly satisfactory audio at low cost. Summary of the Invention

[0004] The present disclosure provides a terminal and method for outputting multi-channel audio using a plurality of audio devices.

[0005] In addition, the present disclosure provides a terminal for providing appropriate arrangement positions of a plurality of audio devices constituting a multi-channel audio system.

[0006] An embodiment of the present disclosure provides a terminal for outputting multi-channel audio using multiple audio devices, the terminal including: a camera; a communication unit connected to multiple first audio devices; and a processor configured to obtain device information about the multiple first audio devices through the communication unit or the camera; configure a multi-channel audio system including at least two second audio devices among the multiple first audio devices based on the device information; and output audio data through the second audio devices based on the audio system information corresponding to the multi-channel audio system.

[0007] The device information may include audio channel information and volume information about each of the plurality of first audio devices.

[0008] The processor may be configured to: obtain, through a camera, first image data including a first audio device; specify the first audio device included in the first image data; and obtain device information corresponding to the specified first audio device.

[0009] The audio system information may include audio channel information of a multi-channel audio system as well as audio channel allocation information, volume information, and arrangement position information about the second audio device.

[0010] The processor may be configured to: obtain second image data of the listening space of the multi-channel audio system through a camera; and determine the arrangement position information based on the second image data.

[0011] The processor may be configured to: obtain first image data including a first audio device through a camera; specify the first audio device included in the first image data; and obtain device information corresponding to the specified first audio device.

[0012] The audio system information may include audio channel information and audio channel allocation information, volume information, and arrangement position information of the second audio device of the multi-channel audio system.

[0013] The processor may be configured to: obtain second image data of the listening space of the multi-channel audio system through a camera; and determine the arrangement position information based on the second image data.

[0014] The terminal may further include an input unit, wherein the processor may be configured to: obtain listening position information in the second image data from the user through the input unit; and determine the arrangement position information based on the listening position information.

[0015] The terminal may further include a display unit, wherein the processor may be configured to output the arrangement position information of the second audio device through the display unit.

[0016] The processor may be configured to: when the number of channels of the audio data is less than the number of channels of the audio channel information, perform upmixing on the audio data; and output the upmixed audio data through the second audio device.

[0017] The communication unit may be configured to receive the upmixed audio data corresponding to the audio data from an upmixing server.

[0018] The processor may be configured to: determine a sound effect based on the audio characteristics of the audio data; and output the audio data by reflecting the sound effect to the audio data through the second audio device.

[0019] The sound effect may include at least one of an equalizer or stereo.

[0020] The processor may be configured to analyze the audio characteristics corresponding to the audio data using an audio analysis model including an artificial neural network.

[0021] In addition, an embodiment of the present disclosure provides a method for outputting multi-channel audio using a plurality of audio devices, the method including the following steps: connecting to a plurality of first audio devices through a communication unit; obtaining device information about the plurality of first audio devices through the communication unit or a camera; configuring a multi-channel audio system including at least two second audio devices among the plurality of first audio devices in consideration of the device information; and outputting audio data through the second audio device based on audio system information corresponding to the multi-channel audio system.

[0022] In addition, an embodiment of the present disclosure provides a recording medium recording a method for outputting multi-channel audio using a plurality of audio devices, the method including the following steps: connecting to a plurality of first audio devices through a communication unit; obtaining device information about the plurality of first audio devices through the communication unit or a camera; configuring a multi-channel audio system including at least two second audio devices among the plurality of first audio devices in consideration of the device information; and outputting audio data through the second audio device based on audio system information corresponding to the multi-channel audio system.

[0023] According to various embodiments of the present disclosure, a multi-channel audio system that is not supported by individual audio devices can be configured by combining a plurality of audio devices.

[0024] In addition, according to various embodiments of the present disclosure, the arrangement position suitable for the audio channels assigned to individual audio devices can be automatically determined and provided to the user.

[0025] In addition, according to various embodiments of the present disclosure, even when the audio device does not originally support upmixing, upmixed audio data can be output through the audio device.

[0026] Furthermore, according to various embodiments of the present disclosure, audio data can be output by applying sound effects suitable for the audio characteristics of the audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a diagram showing a multi-channel audio output system according to an embodiment of the present disclosure.

[0028] Figure 2 is a block diagram showing a terminal for outputting multi-channel audio using a plurality of audio devices according to an embodiment of the present disclosure.

[0029] Figure 3 is a block diagram showing an audio device according to an embodiment of the present disclosure.

[0030] Figure 4 is a flowchart showing a method for outputting multi-channel audio using a plurality of audio devices according to an embodiment of the present disclosure.

[0031] Figure 5 FIG. Figure 5 is a diagram illustrating an example of obtaining device information of an audio device using a camera of a terminal according to an embodiment of the present disclosure.

[0032] Figure 6 is a diagram illustrating a configuration according to an embodiment of the present disclosure Figure 4 FIG. Figure 4 is a flowchart illustrating an example of a multi-channel audio system shown in FIG. Figure 4 .

[0033] Figure 7 FIG. Figure 7 is a diagram illustrating an example of outputting audio system information in a terminal according to an embodiment of the present disclosure.

[0034] Figure 8 is a diagram illustrating an example of outputting audio data through a Figure 4 FIG. Figure 4 is a flowchart illustrating an example of outputting audio data through the multi-channel audio system shown in FIG. Figure 4 .

[0035] Figure 9 FIG. Figure 9 is a diagram illustrating an example of upmixing audio data according to an embodiment of the present disclosure.

[0036] Figure 10 is a diagram illustrating an example of outputting audio data through a Figure 4 FIG. Figure 4 is a flowchart illustrating an example of outputting audio data through the multi-channel audio system shown in FIG. Figure 4 .

[0037] Figure 11 and Figure 12 FIG. Figure 12 is a diagram illustrating a multi-channel audio output system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, and regardless of the reference numerals, the same or similar components are assigned the same reference numerals, and thus redundant descriptions thereof are omitted. The suffixes “module” and “unit” of the components used in the following description are assigned or mixed in consideration of the ease of writing the specification, and have no unique meaning or function. In the following description, detailed descriptions of well-known functions or configurations will be omitted because they will obscure the present invention in unnecessary detail. In addition, the accompanying drawings are used to facilitate the understanding of the embodiments disclosed herein, but the technical idea of the present disclosure is not limited thereto. It will be understood that the present disclosure includes all modifications, equivalents, and alternatives falling within the spirit and scope of the various embodiments of the present disclosure.

[0039] It will be understood that although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0040] It will be understood that when an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or can be connected or coupled to the other element with an intermediate element therebetween. On the other hand, it will be understood that when an element is "directly connected" or "directly coupled" to another element, there is no intermediate element therebetween. Examples of the word "can" used in this disclosure are synonymous with the term "able to" and include its definition.

[0041] The terminal 100 can be implemented as a fixed device or a movable device, such as a TV, a projector, a mobile phone, a smart phone, a desktop computer, a notebook computer, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a tablet PC, a wearable device, a set-top box (STB), a DMB receiver, a radio, a washing machine, a refrigerator, a digital signage, a robot, and a vehicle.

[0042] Figure 1 FIG. is a diagram showing a multi-channel audio output system 1 according to an embodiment of the present disclosure.

[0043] Referring to Figure 1 , the multi-channel audio output system 1 can include a terminal 100, a plurality of audio devices 200, an upmix server 300, and a content provider 400.

[0044] The terminal 100 can refer to a device that is connected to the audio device 200 using wireless communication technology, outputs sound through the audio device 200, and controls the audio device 200. Since the terminal 100 provides an audio signal or audio data to the audio device 200, the terminal 100 can be regarded as an audio signal providing device or an audio data providing device.

[0045] Even in the case where one or more of the plurality of audio devices originally only support mono audio, the terminal 100 can output multi-channel audio through the plurality of audio devices 200. For example, the terminal 100 can output multi-channel audio by sending audio signals of different channels to the audio device 200. In this case, the audio devices 200 that output multi-channel audio can be grouped together and referred to as a single multi-channel audio system. That is, the plurality of audio devices 200 can operate as a single audio system, and such an audio system can operate as a single audio device including a plurality of different sound output units (for example, each audio device can be assigned to a different audio channel). For example, even if two audio devices 200 each only include one sound output unit, one of the two audio devices 200 outputs a left-channel sound corresponding to a left-channel audio signal, and the other of them can output a right-channel sound corresponding to a right-channel audio signal. Therefore, the two audio devices 200 can operate together as a single stereo audio device.

[0046] The terminal 100 can use wireless communication technology to send and receive pairing signals, positioning signals, audio signals, etc. with the audio device 200. The pairing signal can refer to the signal sent and received to connect the audio device 200 to the terminal 100. The positioning signal can refer to the signal sent and received to measure the relative position between the terminal 100 and the audio device 200. The audio signal can refer to the signal sent by the terminal 100 to the audio device 200 to output audio through the audio device 200.

[0047] The terminal 100 can use wired / wireless communication technology to receive content data including audio data from the content provider 400, and can send audio data corresponding to the received audio data to the audio device 200. Alternatively, the terminal 100 can send audio data corresponding to the audio data stored in the memory 170 to the audio device 200.

[0048] The audio device 200 can use wireless communication technology to connect to the terminal 100, and can output audio based on the audio data received from the terminal 100. To this end, the audio device 200 can include at least one sound output unit or speaker.

[0049] The audio device 200 can include one or more sound output units 340. The audio device 200 can output audio of the same channel through each sound output unit 340, or can output audio of different channels. For example, the audio device 200 can include two sound output units, such as headphones or neckband headphones. In this case, the audio device 200 can output left-channel audio and right-channel audio through each sound output unit.

[0050] In an embodiment, when multiple audio devices 200 operate together as a single group, one wireless acoustic device can be used as the main wireless acoustic device, and another wireless acoustic device can be used as the slave wireless acoustic device. In this case, even if the terminal 100 is only connected to the main wireless acoustic device, audio can be output from the slave wireless acoustic device through the main wireless acoustic device. In addition, the terminal 100 can be connected to the slave wireless acoustic device through the main wireless acoustic device, and can output sound from the slave acoustic device.

[0051] In an embodiment, when multiple audio devices 200 operate together as a single group, the terminal 100 can connect all the audio devices 200 and output audio by sending audio data to each audio device 200 individually.

[0052] Among the audio devices 200, the audio device connected to the terminal 100 may be referred to as the first audio device 200_1. Among the first audio devices 200_1, the audio devices constituting the multi-channel audio system may be referred to as the second audio devices 200_2. Since the second audio devices 200_2 are also connected to the terminal 100, the second audio devices 200_2 may also be referred to as the first audio devices 200_1, and vice versa.

[0053] The upmixing server 300 may upmix the audio data received from the terminal 100 using wired / wireless communication technology and may provide the upmixed audio data to the terminal 100. Upmixing or audio upmixing may refer to an operation of expanding the number of channels of audio data. For example, the upmixing server 300 may receive two-channel audio data from the terminal 100, upmix the received audio data into 5.1-channel audio data, and send the 5.1-channel audio data to the terminal 100.

[0054] The upmixing server 300 may be configured as a single server, or may be configured as a cloud server or a server cluster including multiple servers.

[0055] The content provider 400 may provide video content data or audio content data to the terminal 100 using wired / wireless communication technology. The video content or audio content data may include audio data.

[0056] The content provider 400 may refer to various content providing devices, such as radio base stations, terrestrial base stations, broadcast satellites, and content data servers.

[0057] Figure 2 is a block diagram of the terminal 100 that outputs multi-channel audio using multiple audio devices according to an embodiment of the present disclosure.

[0058] Referring to Figure 2 , the terminal 100 may include a communication unit 110, an input unit 120, a learning processor 130, a sensing unit 140, an output unit 150, a memory 170, and a processor 180.

[0059] The communication unit 110 may also be referred to as a communication modem or a communication circuit.

[0060] The communication unit 110 may use wired / wireless communication technology to send data to and receive data from external devices such as the audio device 200. For example, the communication unit 110 may send and receive sensor information, user input, learning models, control signals, etc. to and from external devices.

[0061] The communication technology used by the communication unit 110 includes Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), 5G, Wireless LAN (WLAN), Wireless Fidelity (Wi-Fi), BLUETOOTH TM , Radio Frequency Identification (RFID), Infrared Data Association (IrDA), ZIGBEE, Near Field Communication (NFC), etc.

[0062] The input unit 120 may be referred to as an input interface.

[0063] The input unit 120 may obtain various types of data.

[0064] The input unit 120 may include a camera 121 for inputting image signals, a microphone 122 for receiving audio signals, a user input unit 123 for receiving information from a user, etc. By regarding the camera 121 or the microphone 122 as a sensor, signals obtained from the camera 121 or the microphone 122 may be referred to as sensing data or sensor information.

[0065] Voice data or image data collected by the input unit 120 may be analyzed and processed into a user control command.

[0066] The input unit 120 may obtain training data for model learning, input data to be used to obtain output using a learning model, etc. The input unit 120 may obtain raw input data. In this case, the processor 180 or the learning processor 130 may extract input features by preprocessing the input data.

[0067] The camera 121 processes image frames, such as still images or moving images obtained by an image sensor in a video call mode or a capturing mode, and the processed image frames may be displayed on the display unit 151 or stored in the memory 170.

[0068] The microphone 122 processes the external sound signal into electrical voice data. The processed voice data can be used in various ways according to the function being executed (or the application being run) by the terminal 100. On the other hand, various noise cancellation algorithms for canceling noise generated in the process of receiving the external sound signal can be applied to the microphone 122.

[0069] The user input unit 123 is configured to receive information from a user. When information is input through the user input unit 123, the processor 180 may control the operation of the terminal 100 corresponding to the input information.

[0070] The user input unit 123 may include mechanical input means (e.g., buttons, dome switches, rollers, rocking switches, etc. located on the front / back or side of the terminal 100) and touch input means. For example, the touch input means may include virtual keys, soft keys, or visual keys displayed on the touch screen and processed by software, or may include touch keys provided on parts other than the touch screen.

[0071] The learning processor 130 may use training data to learn a model including an artificial neural network. The learned artificial neural network may be referred to as a learning model. The learning model may be used to infer a result value for new input data other than the training data, and the inferred value may be used as a basis for determining to perform a specific operation.

[0072] The learning processor 130 may perform AI processing together with the learning processor of the artificial intelligence server.

[0073] The learning processor 130 may include a memory integrated or implemented in the terminal 100. Alternatively, the learning processor 130 may be implemented using the memory 170, an external memory directly connected to the terminal 100, or a memory maintained in an external device.

[0074] The sensing unit 140 may be referred to as a sensor unit or a sensor.

[0075] The sensing unit 140 may use various sensors to obtain at least one of the internal information of the terminal 100, the surrounding environment information of the terminal 100, or user information.

[0076] The sensors included in the sensing unit 140 include a proximity sensor, an illuminance sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor, a microphone, a lidar, a radar, etc.

[0077] The output unit 150 may be referred to as an output interface.

[0078] The output unit 150 may generate an output related to vision, audition, or touch. The output unit 150 may include at least one of a display unit 151, a sound output unit 152, a haptic module 153, or an optical output unit 154.

[0079] The display unit 151 displays (outputs) the information processed by the terminal 100. For example, the display unit 151 may display the execution screen information of an application driven in the terminal 100 or user interface (UI) and graphical user interface (GUI) information according to the execution screen information.

[0080] The display unit 151 may implement a touch screen by forming a mutual layer structure with the touch sensor or forming integrally with the touch sensor. The touch screen may be used as a user input unit 123 that provides an input interface between the terminal 100 and the user, and may also provide an output interface between the terminal 100 and the user.

[0081] The sound output unit 152 may output audio data received from the communication unit 110 or stored in the memory 170 in a call signal reception, call mode, recording mode, voice recognition mode, broadcast reception mode, etc. The sound output unit 152 may include at least one of a receiver, a speaker, or a buzzer.

[0082] The haptic module 153 generates various haptic effects that can be felt by the user. A representative example of the haptic effect generated by the haptic module 153 may be vibration.

[0083] The optical output unit 154 uses the light of the light source of the terminal 100 to output a signal for notifying the occurrence of an event. Examples of events generated in the terminal 100 may include message reception, call signal reception, missed call, alarm, daily notification, email reception, information reception through an application, etc.

[0084] The memory 170 may store data supporting various functions of the terminal 100. For example, the memory 170 may store input data obtained from the input unit 120, learning data, learning models, learning histories, etc.

[0085] The processor 180 may control the overall operation of the terminal 100.

[0086] The image signal processed by the processor 180 for image processing may be input to the display unit 151 and displayed as an image corresponding to the image signal. In addition, the image signal processed by the processor 180 for image processing may be input to an external output device through the external device interface unit.

[0087] The audio signal processed by the processor 180 may be output through the sound output unit 152. In addition, the audio signal processed by the processor 180 may be input to an external output device through the external device interface unit.

[0088] The processor 180 may determine at least one executable operation of the terminal 100 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. In addition, the processor 180 may execute the determined operation by controlling the elements of the terminal 100. To this end, the processor 180 may request, retrieve, receive, or utilize the data of the learning processor 130 or the memory 170, and may control the elements of the terminal 100 to execute a predicted operation or an operation determined to be desirable among at least one executable operation.

[0089] When the interconnection of an external device is required to perform a determined operation, the processor 180 may generate a control signal for controlling the corresponding external device and transmit the generated control signal to the corresponding external device.

[0090] The processor 180 may obtain intent information for a user input and may determine a user's request based on the obtained intent information.

[0091] The processor 180 may use at least one of a speech-to-text (STT) engine for converting a voice input into a string or a natural language processing (NLP) engine for obtaining intent information of a natural language to obtain intent information corresponding to the user input. At least one of the STT engine or the NLP engine may be configured as an artificial neural network, at least a part of which learns according to a machine learning algorithm. At least one of the STT engine or the NLP engine may learn through the learning processor 130 of the terminal, may learn through the learning processor of the artificial intelligence server, or may learn through its distributed processing.

[0092] The processor 180 may collect historical information including the operation content or user feedback of the operation of the terminal 100, and may store the collected historical information in the memory 170 or the learning processor 130, or may transmit the collected historical information to the artificial intelligence server or the like. The collected historical information may be used to update the learning model.

[0093] The processor 180 may control at least some elements of the terminal 100 to drive an application program stored in the memory 170. In addition, the processor 180 may drive an application program by combining two or more elements included in the terminal 100.

[0094] On the other hand, Figure 1 The illustrated terminal 100 is only one embodiment of the present disclosure, and thus Figure 1 Some of the illustrated elements may be integrated, added, or omitted according to the specifications of the terminal 100 to be actually implemented.

[0095] In an embodiment, two or more elements may be integrated into one element, or one element may be divided into two or more elements. In addition, the functions provided by each block are for describing the embodiments of the present disclosure, and the specific operations or devices do not limit the scope of the present disclosure.

[0096] Figure 3 is a block diagram of an audio device 200 according to an embodiment of the present disclosure.

[0097] Referring to Figure 3 , the audio device 200 may include a communication unit 210, an input unit 220, an output unit 250, a memory 270, and a processor 280.

[0098] The communication unit 210 may also be referred to as a communication modem or communication circuit.

[0099] The communication unit 210 may use wired / wireless communication technologies to send data to and receive data from an external device such as the terminal 100. For example, the communication unit 110 may send and receive sensor information, user input, learning models, control signals, etc. to and from an external device.

[0100] The communication technologies used by the communication unit 210 include Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), 5G, Wireless Local Area Network (WLAN), Wi-Fi, Bluetooth TM , Radio Frequency Identification (RFID), Infrared Data Association (IrDA), ZigBee, Near Field Communication (NFC), etc.

[0101] The communication unit 210 may receive an FM radio signal or an AM radio signal from a radio base station.

[0102] The input unit 220 may be referred to as an input interface.

[0103] The input unit 220 may include a microphone 222 for receiving an audio signal, a user input unit 223 for receiving information from a user, etc. By regarding the microphone 222 as a sensor, the signal obtained from the microphone 222 may be referred to as sensed data or sensor information.

[0104] The voice data or image data collected by the input unit 220 may be analyzed and processed into a user control command.

[0105] The microphone 222 processes an external sound signal into electrical voice data. The processed voice data may be used in various ways according to the function being performed by the terminal 200 (or the application being run). On the other hand, various noise cancellation algorithms for eliminating noise generated during the reception of an external sound signal may be applied to the microphone 222.

[0106] The user input unit 223 is configured to receive information from a user. When information is input through the user input unit 223, the processor 280 may control the operation of the terminal 200 corresponding to the input information.

[0107] The user input unit 223 may include mechanical input means (e.g., buttons, dome switches, rollers, rocking switches, etc. located on the front / back or side of the terminal 200) and touch input means. For example, the touch input means may include virtual keys, soft keys, or visual keys displayed on a touch screen and processed by software, or may include touch keys provided on a part other than the touch screen.

[0108] The output unit 250 may be referred to as an output interface.

[0109] The output unit 250 may generate outputs related to vision, audition, or touch. The output unit 250 may include at least one of a display unit 151, a sound output unit 152, a haptic module 153, or an optical output unit 154.

[0110] The display unit 251 displays (outputs) the information processed by the terminal 200. For example, the display unit 251 may display execution screen information of an application driven in the terminal 200 or user interface (UI) and graphical user interface (GUI) information based on the execution screen information.

[0111] The display unit 251 may be implemented as a touch screen by forming a mutual layer structure with a touch sensor or integrally forming with the touch sensor. The touch screen may be used as a user input unit 223 that provides an input interface between the terminal 200 and a user, and may also provide an output interface between the terminal 100 and the user.

[0112] The sound output unit 252 may output audio data received from the communication unit 210 or stored in the memory 270 in a call signal reception, call mode, recording mode, voice recognition mode, broadcast reception mode, etc. The sound output unit 252 may include at least one of a receiver, a speaker, or a buzzer.

[0113] The sound output unit 252 may include a diaphragm, an acoustic coil, and a permanent magnet that forms a magnetic field around the acoustic coil. When power is applied to the acoustic coil, an electromagnetic force may be generated within the magnetic field formed by the permanent magnet to move the acoustic coil, and the diaphragm may vibrate according to the movement of the acoustic coil to output sound.

[0114] The memory 270 may store data that supports various functions of the audio device 200. For example, the memory 270 may store input data obtained from the input unit 220, audio data received from the terminal 100, etc.

[0115] The processor 280 may control the overall operation of the audio device 200.

[0116] Figure 4 is a flowchart showing a method of outputting multi-channel audio using multiple audio devices according to an embodiment of the present disclosure.

[0117] Refer to Figure 4 , the processor 180 of the terminal 100 is connected to a plurality of first audio devices 200_1 through the communication unit 110 (S401).

[0118] Since the audio device 200 connected to the terminal 100 is referred to as the first audio device 200_1, the above step S401 can be considered as follows: The processor 180 of the terminal 100 is connected to a plurality of audio devices 200 through the communication unit 110.

[0119] The terminal 100 and the first audio device 200_1 can be connected by pairing with each other. To this end, the first audio device 200_1 can send a pairing request signal (or a connection request signal), and the terminal 100 can establish a connection with the first audio device 200_1 that sends the pairing request signal by scanning the device that sends the pairing request signal.

[0120] The processor 180 of the terminal 100 obtains device information about a plurality of first audio devices 200_1 through the communication unit 110 or the camera 121 (S403).

[0121] The processor 180 can receive device information about the first audio device 200_1 from the first audio device 200_1 through the communication unit 110. Specifically, the terminal 100 can receive device information from the first audio device 200_1 that has been connected or the first audio device 200_1 that sends a connection request signal to establish a connection.

[0122] Alternatively, the processor 180 can obtain first image data including the first audio device 200_1 through the camera 121, can specify the first audio device 200_1 included in the first image data, and can obtain device information about a specific first audio device 200_1. To this end, the processor 180 can specify the first audio device 200_1 from the first image data using an object recognition model, and can receive device information about the recognized first audio device 200_1 through the Internet or an external server.

[0123] The device information may include a device name, a device model name, a device serial number, audio channel information, sound level information, equalizer information, etc. The audio channel information may refer to the number of audio channels that the audio device 200 can output and the configuration information of the audio channels. In order to output multi-channel audio using a plurality of audio devices 200, the processor 180 can obtain the audio channel information about the first audio device 200_1.

[0124] Although Figure 4 It is shown that the step S403 of obtaining device information about the first audio device 200_1 is performed after the step S401 of connecting to the first audio device 200_1, but the present disclosure is not limited thereto. That is, according to an embodiment, the step S401 of connecting the terminal 100 to the first audio device 200_1 and the step S403 of obtaining device information about the first audio device 200_1 can be performed simultaneously, or can be performed in the reverse order.

[0125] The processor 180 of the terminal 100 configures a multi-channel audio system including at least two second audio devices 200_2 among a plurality of first audio devices 200_1 by considering device information (S405).

[0126] The processor 180 may determine the audio channels of the respective first audio devices 200_1 by considering the device information of the plurality of connected first audio devices 200_1, and may configure a multi-channel audio system including at least two second audio devices 200_2 among the plurality of first audio devices 200_1 based on this.

[0127] The processor 180 may generate audio system information while configuring the multi-channel audio system. The audio system information may include audio channel information of the multi-channel audio system, audio channel allocation information regarding each second audio device 200_2, volume information, layout position information, and the like.

[0128] In addition, the processor 180 may output the audio system information regarding the configured multi-channel audio system through the output unit 150.

[0129] The processor 180 may output the second audio devices 200_2 constituting the multi-channel audio system, the audio channel allocation information regarding each second audio device 200_2, the volume information regarding each second audio device 200_2, or the layout position information regarding each second audio device 200_2 based on the audio system information.

[0130] The processor 180 outputs the layout position information regarding each second audio device 200_2 through the output unit 150, thereby guiding the user to arrange the second audio device 200_2 at a position suitable for the configured multi-channel audio system.

[0131] In addition, the processor 180 may inquire of the user whether to configure a multi-channel audio system through the output unit 150, and may configure or not configure the multi-channel audio system based on the user input.

[0132] The processor 180 of the terminal 100 outputs audio data through the multi-channel audio system (S407).

[0133] The processor 180 may output the audio data stored in the memory 170 or the audio data received from the content provider 300 through the second audio devices 200_2 constituting the multi-channel audio system.

[0134] As Figure 8 shown, the processor 180 may perform upmixing on the audio data based on the audio channel information of the multi-channel audio system, and may output the upmixed audio data through the multi-channel audio system. Additionally, as Figure 10As shown, the processor 180 may determine sound effects in consideration of the audio characteristics of audio data, and may output the upmixed audio data to which the sound effects are applied through a multi-channel audio system. Additionally, the processor 180 may upmix the audio data, apply sound effects thereto, and output the resulting data through a multi-channel audio system. When performing both the upmixing of the audio data and the application of the sound effects, it does not matter which one is performed first.

[0135] Figure 4 The step sequence shown is merely an example, and the present disclosure is not limited thereto. That is, Figure 4 the order of some of the steps shown may be changed and executed. Additionally, Figure 4 the order of some of the steps shown may be executed in parallel. Additionally, only Figure 4 some of the steps shown may be executed.

[0136] Figure 5 FIG. is a diagram showing an example of obtaining device information of an audio device using a camera of a terminal.

[0137] Referring to Figure 5 , the terminal 510 may obtain image data (or first image data) 531 including the audio device 520 through the camera, may specify the audio device 520 by recognizing the audio device 520 included in the obtained image data, and may obtain the device information 532 of the specified audio device 520.

[0138] The terminal 510 may directly recognize and specify the audio device 520 from the image data, but may send the image data to a separate external server and obtain the identification information of the audio device 520 included in the image data from the external server.

[0139] Furthermore, the terminal 510 may output the device information 532 of the audio device 520 through the display unit.

[0140] In Figure 5 the example shown, the terminal 510 includes a plurality of display units, but the present disclosure is not limited thereto. That is, the terminal 510 may include one display unit or a plurality of display units.

[0141] Figure 6 FIG. is a flowchart showing an example of step S405 of configuring Figure 4 the multi-channel audio system shown.

[0142] Referring to Figure 6 , the processor 180 of the terminal 100 obtains second image data of the listening space of the multi-channel audio system through the camera 121 (S601).

[0143] The listening space of the multi-channel audio system may refer to the space where multiple second audio devices 200_2 constituting the multi-channel audio system are to be arranged, and may also refer to the space for listening to the audio output through the multiple audio devices 200_2 (e.g., the TV room in a user's home).

[0144] The processor 180 of the terminal 100 obtains listening position information regarding the second image data through the input unit 120 (S603).

[0145] The processor 180 may output the second image data through the display unit 151 and may obtain listening position information regarding the second image data through the user input unit 123. When the display unit 151 is a touch screen capable of touch input, the processor 180 may obtain listening position information regarding the second image data through the display unit 151.

[0146] The processor 180 of the terminal 100 determines audio system information based on the device information and listening position information regarding the first audio device 200_1 (S605).

[0147] The processor 180 may determine the second audio devices 200_2 to configure the multi-channel audio system based on the audio channel information of each first audio device 200_1, and may determine the audio channel allocation information and volume information regarding each second audio device 200_2. Additionally, the processor 180 may determine the arrangement position information regarding the second audio devices 200_2 based on the audio channel allocation information, volume information, and listening position information regarding the second audio devices 200_2.

[0148] The processor 180 of the terminal 100 outputs the audio system information through the display unit 151 (S607).

[0149] The processor 180 may display through the display unit 151 a list of the second audio devices 200_2 constituting the multi-channel audio system, the audio channel information assigned to each second audio device 200_2, and the arrangement positions of the second audio devices 200_2. Specifically, the processor 180 may display the arrangement positions of the second audio devices 200_2 on the second image data.

[0150] Figure 7 It is a diagram showing an example of outputting audio system information in the terminal.

[0151] Refer to Figure 7 , the terminal 710 may obtain image data (or second image data) 731 of the listening space 720 of the multi-channel audio system through the camera, may obtain the listening position information 732 of the image data 731 obtained through the input unit 120, and may configure the multi-channel audio system based on the device information of the audio devices 740_1 to 740_5 and the listening position information 732.

[0152] The image data 731 of the listening space 720 may refer to the image data captured to provide augmented reality (AR).

[0153] The display unit of the terminal 710 may be a touch screen. The terminal 710 may output the second image data 731 to the display unit, and the user may input the listening position information 732 by touching the listening position in the second image data 731 output to the display unit (for example, by touching and selecting a point on the sofa in the room image, etc.).

[0154] As described above, the terminal 710 may configure a multi-channel audio system by determining, among the connected first audio devices, the second audio devices 740_1 to 740_5 that configure a multi-channel audio system, and determining the audio system information including the audio channel allocation information, the volume information, and the arrangement position information for each of the second audio devices 740_1 to 740_5.

[0155] The terminal 710 may output a list 733 of the second audio devices 740_1 to 740_5 that configure a multi-channel audio system through the display unit.

[0156] The terminal 710 may output the second image data 731 and the arrangement position of each of the second audio devices 740_1 to 740_5 that configure a multi-channel audio system through the display unit. In this case, the terminal 710 may display the first image data 531 of the second audio devices 740_1 to 740_5 at the position corresponding to the determined arrangement position information on the second image data 731.

[0157] In Figure 7 the example shown, the terminal 710 includes a plurality of display units, but the present disclosure is not limited thereto. That is, the terminal 710 may include one display unit or a plurality of display units.

[0158] Figure 8 is a flowchart showing an example of step S407 of outputting audio data through Figure 4 the multi-channel audio system shown.

[0159] Referring to Figure 8 , the processor 180 of the terminal 100 determines whether the number of channels of the audio data is less than the number of channels of the multi-channel audio system (S801).

[0160] The processor 180 can check the number of channels of a multi-channel audio system through the audio channel information of the audio system information. For example, when the processor 180 configures six second audio devices 200_2 as a multi-channel audio system, the audio channel information of the audio system information can be 5.1 channels.

[0161] As a result of the determination in step S801, when the number of channels of the audio data is less than the number of channels of the multi-channel audio system, the processor 180 of the terminal 100 performs upmixing on the audio data (S803) and outputs the upmixed audio data through the multi-channel audio system (S805).

[0162] The processor 180 can directly perform upmixing on the audio data according to the number of audio channels of the multi-channel audio system, or can perform upmixing on the audio data through the upmixing server 300. The processor 180 can send the audio data and audio channel information of the multi-channel audio system to the upmixing server 300 through the communication unit 110. The upmixing server 300 can perform upmixing on the audio data based on the received audio data and received audio channel information of the multi-channel audio system, and the processor 180 can receive the upmixed audio data from the upmixing server 300 through the communication unit 110.

[0163] As a result of the determination in step S801, when the number of channels of the audio data is not less than the number of channels of the multi-channel audio system, the processor 180 of the terminal 100 outputs the non-upmixed audio data through the multi-channel audio system (S807).

[0164] Although Figure 8 illustrates an example in which the terminal 100 performs upmixing on the audio data when the number of channels of the audio data is less than the number of channels of the multi-channel audio system, the present disclosure is not limited thereto. In another embodiment, even when the number of channels of the audio data is less than the number of channels of the multi-channel audio system, the terminal 100 may output the audio data through the multi-channel audio system without performing upmixing on the audio data.

[0165] Figure 9 is a diagram showing an example of performing upmixing on audio data.

[0166] Referring to Figure 9 , after configuring the multi-channel audio system, the terminal 910 can output the number (921) of the audio devices 930_1 to 930_5 that constitute the multi-channel audio system and perform upmixing on the information corresponding to the multi-channel audio system.

[0167] In addition, when the terminal 910 configures a multi-channel audio system, or when the terminal 910 outputs audio data through the multi-channel audio system, the terminal 910 can output a notification (922) asking whether to perform upmixing on the audio data according to the multi-channel audio system.

[0168] Figure 10 is a flowchart showing an example of step S407 of outputting audio data through the Figure 4 multi-channel audio system shown.

[0169] Referring to Figure 10 , the processor 180 of the terminal 100 determines whether to apply a sound effect when outputting audio data (S1001).

[0170] The processor 180 may determine whether to apply a sound effect when outputting audio data based on a user input through the input unit 120 or a preset value. The sound effect may include at least one of an equalizer or stereo.

[0171] When it is determined in step S1001 that a sound effect is to be applied, the processor 180 of the terminal 100 determines the sound effect based on the audio characteristics of the audio data (S1003), and outputs the audio data to which the sound effect is applied through the multi-channel audio system (S1005).

[0172] The processor 180 may analyze the audio characteristics of the audio data by considering the frequency distribution of sound waves, voices, background sounds, etc. included in the audio data, and may determine the sound effect based on the analyzed audio characteristics of the audio data.

[0173] The processor 180 may determine the content of the audio data based on the audio characteristics of the audio data, and may determine the sound effect to be applied to the audio data by selecting a preset sound effect corresponding to the determined content.

[0174] The processor 180 may determine a mode flag of the sound effect based on the content of the audio data, and may apply the sound effect according to the determined mode flag. The following [Table 1] shows an example of the mode flag of the sound effect.

[0175] [Table 1]

[0176]

[0177]

[0178] Referring to [Table 1] above, the "Center Boost" stereo sound effect can be applied to audio data corresponding to news or sports that mainly have a voice component, the "Widened" stereo sound effect can be applied to audio data corresponding to stereo music with a large amount of background sound, the "Surround Widened" stereo sound effect can be applied to audio data corresponding to movies, and the no stereo sound effect can be not applied to audio data corresponding to entertainment programs that are difficult to give a stereo effect due to frequent scene changes. The processor 180 can use an audio analysis model configured with an artificial neural network to analyze the audio characteristics of the audio data. For example, the audio analysis model can be a model that determines and outputs the content or genre included in the audio data when the audio data is input, or can be a model that determines and outputs the sound effect or mode flag to be applied to the audio data when the audio data is input.

[0179] The processor 180 can determine the sound effect by directly analyzing the audio data of the audio data, or can determine the sound effect by analyzing the audio data via an external server (e.g., the upmixing server 300, etc.). The processor 180 can send the audio data to the external server through the communication unit 110. The external server can determine the sound effect by analyzing the audio characteristics of the received audio data, and the processor 180 can receive the determined sound effect information from the external server through the communication unit 110.

[0180] The processor 180 can apply the sound effect to the audio data and output the audio data by sending the information about the determined sound effect (e.g., the mode flag) together with the audio data to the second audio device 200_2. Alternatively, the processor 180 can apply the sound effect to the audio data and output the audio data by sending the audio data with the applied sound effect to the second audio device 200_2 after applying the sound effect to the audio data.

[0181] When no sound effect is applied as a result of the determination in step S1001, the processor 180 of the terminal 100 outputs the audio data without the applied sound effect through the multi-channel audio system (S1007).

[0182] Figure 11 and Figure 12 are diagrams showing multi-channel audio output systems 1100 and 1200 according to embodiments of the present disclosure.

[0183] Referring to Figure 11 and Figure 12 According to embodiments of the present disclosure, the multi-channel audio output systems 1100 and 1200 may include a terminal 1110, a plurality of audio devices 1120_1 to 1120_4, display devices 1130 and 1230, and a microphone 1140. In Figure 11 the display device 1130 can be a device such as a TV or a monitor including a display panel. In Figure 12In [the figure], the display device 1230 may be a projector that projects light corresponding to an image onto a wall or screen 1235 through a light source.

[0184] The terminal 1110 may be connected to a plurality of audio devices 1120_1 to 1120_4, may configure a multi-channel audio system by using the connected audio devices 1120_1 to 1120_4, and may output audio data through the plurality of audio devices 1120_1 to 1120_4 that configure the multi-channel audio system. The terminal 1110 may output information about the multi-channel audio system and a notification asking whether upmixing corresponding to the multi-channel audio system can be performed and whether to perform upmixing through a display unit (1111).

[0185] The terminal 1110 may be connected to display devices 1130 and 1230 and output image data through the connected display devices 1130. Additionally, the terminal 1110 may be connected to a microphone 1140 and receive audio data through the connected microphone 1140.

[0186] The terminal 1110 may configure a multi-channel audio system by using the plurality of audio devices 1120_1 to 1120_4 and send audio system information of the multi-channel audio system to the connected display devices 1130 and 1230. Additionally, the terminal 1110 may determine arrangement position information of each of the audio devices 1120_1 to 1120_4 in consideration of the audio channels assigned to each of the plurality of audio devices 1120_1 to 1120_4, and may output the determined arrangement position information through the display unit. Therefore, even after the display devices 1130 and 1230 are disconnected from the terminal 1110, the display devices 1130 and 1230 may still output multi-channel audio through the plurality of audio devices 1120_1 to 1120_4 that constitute the multi-channel audio system.

[0187] In an embodiment, the multi-channel audio output systems 1100 and 1200 may provide a home karaoke service. To this end, the terminal 1110 may execute a karaoke application, may display a karaoke application screen through the display devices 1130 and 1230, may output the sound of the karaoke application through the plurality of audio devices 1120_1 to 1120_4, and may receive the voice of the user through the microphone 1140.

[0188] In an embodiment, the multi-channel audio output systems 1100 and 1200 may provide a home theater service. To this end, the terminal 1110 may display an image of video content through the display devices 1130 and 1230, and may output the audio of the video content through the plurality of audio devices 1120_1 to 1120_4.

[0189] In addition, the terminal 1110 may receive content including audio data from a separate content provider 400. Additionally, the terminal 1110 may perform upmixing on the audio data through a separate upmixing server 300.

[0190] According to an embodiment of the present disclosure, the above method may be implemented on a medium having a program recorded thereon using computer-readable code. The computer-readable medium includes any type of recording device that stores data readable by a computer system. Examples of the computer-readable medium include a hard disk drive (HDD), a solid state drive (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0191] Cross-reference to related applications

[0192] This application is a national phase of PCT International Application No. PCT / KR2020 / 011206, filed on August 21, 2020, which is hereby incorporated by reference in its entirety.

Claims

1. A terminal for outputting multi-channel audio using multiple audio devices, the terminal comprising: A camera; An input unit; A display unit; A communication unit connected to multiple first audio devices; And A processor configured to: Obtain device information about the multiple first audio devices through the communication unit or the camera; Configure a multi-channel audio system including at least two second audio devices selected from among the multiple first audio devices based on the device information; and Output audio data through the at least two second audio devices based on audio system information corresponding to the multi-channel audio system, Wherein the audio system information includes arrangement position information about the second audio devices, Wherein the processor is further configured to: Obtain first image data including the first audio devices; Designate the first audio devices included in the first image data; Obtain device information corresponding to the designated first audio devices; Obtain second image data of a listening space for the multi-channel audio system through the camera; Obtain listening position information in the second image data from the user through the input unit; Determine the arrangement position information based on the listening position information; and Output the arrangement position information about the second audio devices through the display unit, Wherein the arrangement position information guides the user to arrange the second audio devices at positions suitable for the configured multi-channel audio system.

2. The terminal according to claim 1, wherein The device information includes audio channel information and volume information of each of the multiple first audio devices.

3. The terminal according to claim 2, wherein The audio system information includes audio channel information of the multi-channel audio system, audio channel allocation information, and volume information about the at least two second audio devices.

4. The terminal according to claim 3, wherein, The processor is further configured to: When the number of channels of the audio data is less than the number of channels of the multi-channel audio system, perform upmixing on the audio data to generate upmixed audio data; and Output the upmixed audio data through the at least two second audio devices.

5. The terminal according to claim 4, wherein, The communication unit is configured to receive upmixed audio data corresponding to the audio data from an upmix server.

6. The terminal according to claim 2, wherein The processor is further configured to: Determine a sound effect based on the audio characteristics of the audio data or a user input selection; and Output the audio data through the at least two second audio devices by reflecting the sound effect.

7. The terminal according to claim 6, wherein, The sound effect includes at least one of an equalizer or stereo.

8. The terminal according to claim 6, wherein The processor is further configured to analyze audio characteristics corresponding to the audio data using an audio analysis model including an artificial neural network.

9. A method for outputting multi-channel audio using multiple audio devices, the method comprising the following steps: Connect to multiple first audio devices through a communication unit; Obtain device information about the multiple first audio devices through the communication unit or a camera; Configure a multi-channel audio system including at least two second audio devices selected from among the multiple first audio devices based on the device information; and Output audio data through the at least two second audio devices based on audio system information corresponding to the multi-channel audio system, wherein the audio system information includes arrangement position information about the second audio devices, wherein the method further comprises: obtain first image data including the first audio device; designate the first audio device included in the first image data; obtain device information corresponding to the designated first audio device; obtain second image data of the listening space for the multi-channel audio system through the camera; obtain listening position information in the second image data from the user through the input unit; determine the arrangement position information based on the listening position information; and output the arrangement position information about the second audio devices through the display unit, wherein the arrangement position information guides the user to arrange the second audio devices at positions suitable for the configured multi-channel audio system.

10. A non-transitory computer-readable recording medium having computer-executable instructions recorded thereon, which when executed by a computer, perform the method according to claim 9.

11. A method for controlling a terminal device to configure a plurality of first audio devices as a multi-channel audio system, the method comprising the steps of: wirelessly connect to the plurality of first audio devices via a wireless communication unit of the terminal device; receive device information about the plurality of first audio devices; determine, based on the device information via a controller in the terminal device, at least two second audio devices as a group from among the plurality of first audio devices to provide a multi-channel audio system; obtain an image of the listening space including the at least two second audio devices via a camera in the terminal device; determine placement information for arranging the at least two second audio devices at different positions within the listening space based on the device information; display the placement information on a display of the terminal device; receive audio data; send first audio channel data corresponding to the audio data to one of the at least two second audio devices; and send second audio channel data corresponding to the audio data to another of the at least two second audio devices.

12. The method according to claim 11, the method further comprising the steps of: display the image of the listening space on a display of the terminal device; receive a user input selection of a point within the image; determine placement information for arranging the at least two second audio devices at different positions within the listening space based on the point selected by the user input; and display the placement information on the display of the terminal device.

13. The method according to claim 11, the method further comprising the steps of: in response to the number of audio channels in the audio data being less than the number of audio channels available for output in the multi-channel audio system, perform upmixing on the audio to generate upmixed audio data; and Output the upmixed audio data through the one second audio device and the other second audio device.

14. The method according to claim 11, wherein, Each of the one second audio device and the other second audio device is a mono audio device that can only output one audio channel at a time, wherein, by outputting left channel data through the one second audio device and right channel data through the other second audio device, stereo sound is output by the multi-channel audio system including the one second audio device and the other second audio device.

Images