Electronic device and operating method of the same

Abstract

An electronic device is disclosed. The electronic device includes: an image data receiving circuit configured to receive input image data associated with an image captured by a camera from a camera; a voice data receiving circuit configured to receive input voice data associated with the voices of speakers; a memory configured to store transformation parameters for projecting a spatial coordinate system onto an image coordinate system on an image; and a processor that determines the spatial position of a speaker from the input voice data, converts it to the image position of the speaker on an image, and generates output image data by inserting text associated with the speaker's voice into the input image according to the image position.

Description

Technology Field

[0001] Embodiments of the present invention relate to an electronic device and a method of operating the electronic device. Background Technology

[0002] A microphone is a device that recognizes speech and converts the recognized speech into an electrical signal called a voice signal. When a microphone is placed in a space where multiple speakers are present, such as a conference room or classroom, the microphone receives speech from all of them and generates voice signals associated with their voices. Therefore, when multiple speakers speak simultaneously, it is necessary to separate their voice signals. Furthermore, it is necessary to identify which speaker each separated voice signal is from. Prior art literature

[0003] Korean Published Patent Application No. 10-2016-0091725 (Aug. 3, 2016) The problem to be solved

[0004] The problem that the present invention aims to solve is to provide an electronic device capable of determining the location of a speaker using input voice data and separating the input voice data by speaker, and a method for operating the electronic device.

[0005] The problem that the present invention aims to solve is to provide an electronic device and a method of operating the electronic device that can easily identify the speaker of a voice associated with voice data by determining the location of a speaker terminal, determining the location of the speaker of input voice data, and identifying the speaker terminal existing at a location corresponding to the location of the speaker. means of solving the problem

[0006] An electronic device according to embodiments of the present invention comprises: an image data receiving circuit configured to receive input image data associated with an input image captured by a camera; a voice data receiving circuit configured to receive input voice data associated with the voices of speakers; a memory configured to store transformation parameters for projecting a spatial coordinate system onto an image coordinate system in an image; a processor configured to generate output image data using input voice data and input image data; and an image data output circuit configured to output output image data. The processor generates first spatial position data representing a first spatial position in the spatial coordinate system of a first speaker among the speakers and first text data associated with the voice of the first speaker using input voice data, transforms the first spatial position data into first image position data representing a first image position in the image coordinate system of the first speaker based on transformation parameters, and generates output image data by inserting text associated with the voice of the first speaker into the input image according to the first image position based on input image data, first image position data, and first text data.

[0007] A method of operating an electronic device according to embodiments of the present invention comprises the steps of: receiving input image data associated with an input image captured by a camera from a camera; receiving input voice data associated with the voices of speakers; generating, using the input voice data, first spatial position data representing a first spatial position in a spatial coordinate system of a first speaker among the speakers and first text data associated with the voice of the first speaker; converting the first spatial position data into first image position data representing a first image position in a video coordinate system of the image of the first speaker; generating output image data by inserting text associated with the voice of the first speaker into the input image according to the first image position based on the input image data, the first image position data, and the first text data; and outputting the output image data. Effects of the invention

[0008] The electronic device and the method of operation according to embodiments of the present invention have the effect of determining the spatial location of a speaker from the speaker's voice, converting the speaker's spatial location into the speaker's image location on an image where the speaker is displayed, and generating an image in which text associated with the speaker's voice is displayed at a location corresponding to the speaker's image location.

[0009] Accordingly, users can easily identify who the speaker is that uttered the voice corresponding to the text by looking at the location of the text displayed in the video. Brief explanation of the drawing

[0010] FIG. 1 shows a system including an electronic device according to embodiments of the present invention. FIG. 2 shows an electronic device according to embodiments of the present invention. FIGS. 3 to 5 are drawings for explaining the operation of an electronic device according to embodiments of the present invention. FIG. 7 is a drawing for explaining the operation of an electronic device according to embodiments of the present invention. FIG. 8 is a flowchart for explaining the operation of an electronic device according to embodiments of the present invention. Specific details for implementing the invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the attached drawings.

[0012] FIG. 1 illustrates a system including an electronic device according to embodiments of the present invention. Referring to FIG. 1, the system (10) may include a plurality of microphones (100), an electronic device (200), a camera (300), and a display device (400).

[0013] A system (10) according to embodiments of the present invention receives input voice data associated with the voices of speakers (SPK1 to SPK4) and can generate voice data associated with the voices of each speaker (SPK1 to SPK4). That is, the system (10) can separate voice data associated with the voices of each speaker (SPK1 to SPK4) from the input voice data associated with the voices of speakers (SPK1 to SPK4). According to embodiments, the system (10) can determine the location of the speakers (SPK1 to SPK4) using the input voice data associated with the voices of speakers (SPK1 to SPK4) and generate voice data corresponding to the determined location of the speakers (SPK1 to SPK4).

[0014] Additionally, the system (10) according to embodiments of the present invention may display voice data separated according to speakers (SPK1 to SPK4) as text at the position of the corresponding speaker (SPK1 to SPK4) in an image (IMG) on a display device (400) where the speaker (SPK1 to SPK4) appears. For example, the system (10) may display first voice data associated with the first voice of the first speaker (SPK1) as text at the position of the first speaker (SPK1) in the image (IMG). For example, the text may appear in the form of a speech bubble.

[0015] Speakers (SPK1~SPK4) may be located at each position (P1~P4). According to embodiments, speakers (SPK1~SPK4) located at each position (P1~P4) may pronounce a sound. For example, a first speaker (SPK1) located at the first position (P1) may pronounce a first sound, a second speaker (SPK2) located at the second position (P2) may pronounce a second sound, a third speaker (SPK3) located at the third position (P3) may pronounce a third sound, and a fourth speaker (SPK4) located at the fourth position (P4) may pronounce a fourth sound. Meanwhile, the embodiments of the present invention are not limited to the number of speakers.

[0016] The voices of speakers (SPK1~SPK4) can be received by multiple microphones (100). The multiple microphones (100) can be placed in a space where the voices of speakers (SPK1~SPK4) can be received.

[0017] A plurality of microphones (100) can generate voice signals (VS1 to VSn) associated with voice. According to embodiments, each of the plurality of microphones (100) can measure a pressure change of a medium (e.g., air) caused by voice, generate voice signals (VS1 to VSn) which are electrical signals based on the measured pressure change of the medium, and output the generated voice signals (VS1 to VSn).

[0018] A plurality of microphones (100) can receive the voice of speakers (SPK1~SPK4) located at each position (P1~P4) and convert the voice of the speakers (SPK1~SPK4) into an electrical signal, which is a voice signal. For example, a first microphone can receive the voice of speakers (SPK1~SPK4) and generate a first voice signal associated with the voice of speakers (SPK1~SPK4). The first voice signal generated by the first microphone can correspond to the voice of at least one speaker (SPK1~SPK4).

[0019] A plurality of microphones (100) can output voice signals. According to embodiments, a plurality of microphones (100) can transmit voice signals to an electronic device (200). For example, a plurality of microphones (100) can transmit voice signals to an electronic device (200) via a wired or wireless method.

[0020] A plurality of microphones (100) are configured as beamforming microphones, so that voice can be received from multiple directions. According to embodiments, a plurality of microphones (100) may be spaced apart from each other to form a single microphone array, but embodiments of the present invention are not limited thereto.

[0021] Each of the plurality of microphones (100) may be a directional microphone configured to receive sound from any specific direction, or an omnidirectional microphone configured to receive sound from all directions.

[0022] The electronic device (200) may be a computing device having a computational processing function. According to embodiments, the electronic device (200) may be implemented as a computer, a laptop, a mobile device, a smartphone, or a wearable device, but is not limited thereto. For example, the electronic device (200) may include at least one integrated circuit having a computational processing function.

[0023] The electronic device (200) receives input voice data associated with the voice of the speakers (SPK1~SPK4) and can separate (or generate) output voice data associated with the voice of each speaker (SPK1~SPK4) from the input voice data.

[0024] According to embodiments, the electronic device (200) receives voice signals transmitted from a plurality of microphones (100) and can obtain input voice data associated with the voice of a speaker (SPK1~SPK4) from the voice signals.

[0025] Meanwhile, although the present specification describes the electronic device (200) assuming that it receives voice signals from a plurality of microphones (100) to obtain input voice data associated with the voice of a speaker (SPK1~SPK4), according to embodiments, the electronic device (200) may also receive input voice data associated with the voice of a speaker from an external device.

[0026] According to embodiments, the electronic device (200) and a plurality of microphones (100) may be formed integrally, but are not limited thereto.

[0027] The electronic device (200) can determine the location of the speaker (SPK1~SPK4) (i.e., the location of the sound source) using input voice data associated with the voice of the speaker (SPK1~SPK4). According to embodiments, the electronic device (200) can generate spatial location data indicating the spatial location of the sound source (i.e., the spatial location of the speaker) from the input voice data associated with the voice of the speaker (SPK1~SPK4), based on at least one of the distance between a plurality of microphones (100), the difference between the time points when each of the plurality of microphones (100) receives the voice of the speaker (SPK1~SPK4), and the loudness of the voice of the speaker (SPK1~SPK4).

[0028] In other words, spatial position data represents the position in space within the spatial coordinate system of the speakers (SPK1~SPK4) who uttered each of the input voice data.

[0029] The electronic device (200) can separate input voice data according to the speaker's location (i.e., the location of the sound source) based on spatial location data of the voice.

[0030] Since the location of the sound source can be estimated through the input voice data, each input voice data can correspond to a specific location. At this time, the electronic device (200) can generate output voice data associated with a voice uttered from the speaker's location based on the spatial location data. That is, the output voice data can be associated with a voice uttered from each of the speakers (SPK1~SPK4).

[0031] For example, the electronic device (200) can generate first output voice data associated with a voice spoken at a first location from input voice data using first spatial location data representing a first location, and the first output voice data may be data associated with a voice spoken by a first speaker (SPK1) at the first location.

[0032] For example, when a first speaker (SPK1) and a second speaker (SPK2) speak in temporal overlap, the voices of the first speaker (SPK1) and the second speaker (SPK2) overlap; therefore, each of the voice signals generated by the plurality of microphones (100) corresponds to the overlapped voices of the first speaker (SPK1) and the second speaker (SPK2). Accordingly, the input voice data also includes voice data associated with the voice of the first speaker (SPK1) and voice data associated with the voice of the second speaker (SPK2). As described above, the electronic device (200) can generate spatial location data representing the respective locations of the first speaker (SPK1) and the second speaker (SPK2) from superimposed input voice data associated with the voice of the first speaker (SPK1) and the voice of the second speaker (SPK2), and based on the spatial location data, generate first output voice data associated with the voice of the first speaker (SPK1) and second output voice data associated with the voice of the second speaker (SPK2) from the superimposed input voice data.

[0033] The camera (300) can capture images and generate image data based on the capture results. According to embodiments, the camera (300) can capture a still image or a moving image. For example, the camera (300) can capture the space where the speakers (SPK1~SPK4) are located.

[0034] According to embodiments, the camera (300) may include at least one lens, an image sensor that generates an image signal corresponding to an image input through the at least one lens, and an image signal processor (ISP) that generates image data by processing the image signal generated from the image sensor.

[0035] According to embodiments, for example, the camera (300) may be implemented in the form of a camcorder, smartphone, tablet, laptop, or TV, but embodiments of the present invention are not limited thereto.

[0036] The camera (300) can transmit image data generated according to image capture to an electronic device (200). The image data transmitted to the electronic device (200) may be referred to as input image data.

[0037] A display device (400) can visually display an image (IMG) corresponding to the image data using input image data. According to embodiments, the display device (400) includes a plurality of pixels and a driving circuit for driving the pixels, and the driving circuit can transmit an image signal corresponding to the input image data to the plurality of pixels, and the plurality of pixels can display an image (IMG) by emitting light in response to the image signal.

[0038] Meanwhile, although FIG. 1 shows the camera (300) and the display device (400) connected to each other, the camera (300) and the display device (400) may not be connected to each other. For example, the camera (300) and the display device (400) may be located in different spaces.

[0039] According to embodiments, the display device (400) receives output image data output from the electronic device (200) and can display an image (IMG) based on the output image data.

[0040] FIG. 2 illustrates an electronic device according to embodiments of the present invention. Referring to FIG. 1 and FIG. 2, the electronic device (200) may include an image data receiving circuit (210), a voice data receiving circuit (220), a memory (230), a processor (240), and an image data output circuit (250).

[0041] The image data receiving circuit (210) can receive input image data. According to embodiments, the image data receiving circuit (210) can receive input image data corresponding to an image captured by the camera (300).

[0042] The video data receiving circuit (210) can store input video data in memory (230).

[0043] The voice receiving circuit (220) can receive input voice data associated with the voice of the speaker (SPK1~SPK4). According to embodiments, the voice receiving circuit (220) can receive input voice data associated with the voice of the speaker (SPK1~SPK4) according to a wired communication method or a wireless communication method.

[0044] According to embodiments, the voice receiving circuit (220) includes an analog-to-digital (ADC) converter, receives analog type voice signals (VS1 to VSn) from a plurality of microphones (100), converts the voice signals (VS1 to VSn) into digital type input voice data, and stores the converted input voice data.

[0045] According to embodiments, the voice receiving circuit (220) includes a communication circuit capable of communicating according to a wireless communication method and can receive input voice data through the communication circuit.

[0046] The memory (230) can store data necessary for the operation of the electronic device (200). According to embodiments, the memory (230) may include at least one of a non-volatile memory and a volatile memory.

[0047] According to embodiments, the memory (230) may store transformation parameters for projecting a spatial coordinate system in space onto an image coordinate system within an image captured by the camera (300). That is, the transformation parameters are parameters representing the relationship between the position of a specific object in space and the position of the specific object within an image generated by the object being captured by the camera (300).

[0048] For example, the memory (230) can store transformation parameters corresponding to each of the position of the camera (300) in the spatial coordinate system, the shooting direction (or angle) of the camera (300), the focal length of the camera (300), the principal point of the camera (300), and the asymmetry coefficient of the camera (300).

[0049] The processor (240) can control the overall operation of the electronic device (200). According to embodiments, the processor (240) can generate control commands to control the operation of the wireless signal receiving circuit (210), the voice data receiving circuit (220), the memory (230), and the image data output circuit (250), and transmit the control commands to each of the wireless signal receiving circuit (210), the voice data receiving circuit (220), the memory (230), and the image data output circuit (250).

[0050] The processor (240) may be implemented as an integrated circuit having computation processing capabilities. For example, the processor (240) may include a CPU (central processing unit), MCU (micro controller unit), DSP (digital signal processor), GPU (graphics processing unit), ASIC (application specific integrated circuit), or FPGA (field programmable gate array), but embodiments of the present invention are not limited thereto.

[0051] The processor (240) described in this specification may be implemented with one or more elements. For example, the processor (240) may include a plurality of sub-processors.

[0052] The processor (240) can determine the location of the speaker (SPK1~SPK) (i.e., the location of the sound source of the voice) using input voice data associated with the voice of the speaker (SPK1~SPK4) and generate spatial location data indicating the location of the speaker (SPK1~SPK4). For example, the processor (240) can store the spatial location data in memory (230).

[0053] The processor (240) can generate spatial location data indicating the location of the speaker (SPK1~SPK4) from input voice data associated with the voice of the speaker (SPK1~SPK4) based on at least one of the distance between the electronic device (200) and the plurality of microphones (100), the distance between each of the plurality of microphones (100), the difference between the times when each of the plurality of microphones (100) receives the voice of the speaker (SPK1~SPK4), and the loudness of the voice of the speaker (SPK1~SPK4).

[0054] The position of the speaker (SPK1~SPK4) determined by the electronic device (200) may be a relative position with respect to the electronic device (200). That is, the spatial position data may represent the position of the speaker (SPK1~SPK4) in a spatial coordinate system with respect to the electronic device (200). Alternatively, the position of the speaker (SPK1~SPK4) determined by the electronic device (200) may be an absolute position.

[0055] In this specification, the location of the speaker (SPK1~SPK4) determined by the electronic device (200) may be referred to as a spatial location.

[0056] The processor (240) can separate input voice data according to the speaker's location (i.e., the location of the sound source) based on spatial location data representing the spatial location of the speaker (SPK1~SPK4). For example, the processor (240) can generate output voice data associated with a voice corresponding to the speaker's location based on the input voice data and spatial location data, and can store the output voice data by matching it with the corresponding spatial location data.

[0057] According to embodiments, the processor (240) can generate text data associated with the voice corresponding to the speaker's location based on input voice data and spatial location data. For example, the processor (240) can generate text data by converting output voice data into speech-to-text.

[0058] That is, the output voice data may be data associated with the voice of each speaker (SPK1~SPK4) located at each position (P1~P4), and the text data may be data associated with the text of the voice of each speaker (SPK1~SPK4).

[0059] The processor (240) can convert the spatial position of the speaker (SPK1~SPK4) into the position on image of the speaker (SPK1~SPK4) within the image captured by the camera (300). According to embodiments, the processor (240) can convert the spatial position of the speaker (SPK1~SPK4) in the spatial coordinate system into the position on image by projecting it onto the image coordinate system of the image captured by the camera (300).

[0060] The processor (240) can convert spatial position data representing the spatial position in the spatial coordinate system of the speaker (SPK1~SPK4) into image position data representing the image position in the image coordinate system of the image captured by the camera (300) using the conversion parameters stored in the memory (230).

[0061] The processor (240) can generate output image data using input image data received by the image data receiving circuit (210).

[0062] The processor (240) can generate output image data by inserting content (e.g., text) associated with the voice of the corresponding speaker (SPK1~SPK4) at a position corresponding to the image position of the speaker (SPK1~SPK4) within the image of the input image data. In other words, the processor (240) can generate output image data for displaying content associated with the voice of the corresponding speaker (SPK1~SPK4) at a position corresponding to the image position of the speaker (SPK1~SPK4) within the image of the input image data.

[0063] For example, the processor (240) can generate output image data by inserting content associated with output voice data into the input image data according to the image location indicated by the image location data. For example, the input image data and content can be synthesized according to the image location.

[0064] The video data output circuit (250) can output video data. According to embodiments, the audio data output circuit (250) can output video data to a display device (400) according to a wired communication method or a wireless communication method.

[0065] FIGS. 3 to 6 are drawings for explaining the operation of an electronic device according to embodiments of the present invention. Referring to FIG. 3, spatial positions (P1, P2, P3, P4, O, C) in a spatial coordinate system of speakers (SPK1~SPK4), a plurality of microphones (100), and a camera (300) are shown. The spatial positions (P1, P2, P3, P4, O, C) shown in FIG. 3 are positions based on the plurality of microphones (100). For example, when the plurality of microphones (100) and the electronic device (200) are implemented as a single unit, the spatial positions shown in FIG. 3 become positions based on the electronic device (200).

[0066] The camera (300) can photograph speakers (SPK1~SPK4) and generate input image data corresponding to the captured image (IMG). The display device (400) can receive the input image data and display the captured image (IMG) using the input image data.

[0067] Speakers (SPK1~SPK4) may be displayed in the captured video (IMG). In this case, since the spatial coordinate system serving as the reference in space differs from the image coordinate system serving as the reference in the image, the respective positions of the speakers (SPK1~SPK4) within the image (i.e., image positions) differ from their positions within space (i.e., spatial positions). For example, the first spatial position (P1) and the first image position (Q1) of the first speaker (SPK1) may be different. Therefore, to calculate the image positions of the speakers (SPK1~SPK4) within the image, it is necessary to appropriately transform the spatial positions of the speakers (SPK1~SPK4).

[0068] The electronic device (200) according to embodiments of the present invention has the effect of converting the spatial positions of speakers (SPK1~SPK4) into image positions of speakers (SPK1~SPK4) on an image (IMG), and generating output image data for displaying text associated with the voice of speakers (SPK1~SPK4) at a position corresponding to the image positions of speakers (SPK1~SPK4).

[0069] Referring to Fig. 4, the first speaker (SPK1) utters the first sound "◎◎◎" and the second speaker (SPK2) utters the second sound "☆☆☆".

[0070] The electronic device (200) can receive input voice data (overlapping) associated with a voice in which the first voice “◎◎◎” and the second voice “☆☆☆” are overlapping.

[0071] The processor (240) can generate first spatial location data representing the first spatial location (P1) of the first speaker (SPK1) and second spatial location data representing the second spatial location (P2) of the second speaker (SPK2) using the superimposed input voice data. Meanwhile, although the spatial location (P1 and P2) is depicted as a single point in FIG. 4, according to the embodiments, the spatial location (P1 and P2) may represent a set of one or more points, that is, an area of ​​a certain range.

[0072] Additionally, the processor (240) can use spatial location data to generate first output voice data associated with voice "◎◎◎" spoken at the first spatial location (P1) (i.e., voice of the first speaker (SPK1)) and second output voice data associated with voice "☆☆☆" spoken at the second spatial location (P2) (i.e., voice of the second speaker (SPK2)) from superimposed input voice data. That is, the processor (240) can separate and generate first output voice data associated with voice "◎◎◎" and second output voice data associated with voice "☆☆☆" from input voice data in which the first voice "◎◎◎" and the second voice "☆☆☆" are superimposed. For example, the processor (240) can match and store the first output voice data with the first spatial location data and match and store the second output voice data with the second spatial location data.

[0073] Additionally, according to embodiments, the processor (240) can generate first text data representing a character corresponding to the first voice "◎◎◎" using first output voice data, and can generate second text data representing a character corresponding to the second voice "☆☆☆" using second output voice data.

[0074] The processor (240) can convert spatial location data generated from input voice data into image location data representing a location on an image (IMG). According to embodiments, the processor (240) can convert spatial location data into image location data using conversion parameters stored in memory (230).

[0075] Referring to FIG. 5, the processor (240) can convert spatial position data representing a spatial position in a spatial coordinate system (SCS) into image position data representing an image position in an image coordinate system (ICS).

[0076] The processor (240) can convert a spatial position in a spatial coordinate system (SCS) in space into an image position in an image coordinate system (ICS) in image (IMG). For example, the processor (240) can convert a first spatial position (P(X1,Y1,Z1)) in a spatial coordinate system (SCS) into a first image position (Q1(x1,y1)) in an image coordinate system (ICS).

[0077] According to embodiments, the processor (240) can convert a first spatial position (P(X1,Y1,Z1)) in a spatial coordinate system (SCS) to a first camera position (R1) in a camera coordinate system (CCS), and convert the first camera position (R1) in the camera coordinate system (CCS) to a first image position (Q1(x1,y1)) in an image coordinate system (ICS).

[0078] For example, the processor (240) can convert a first spatial position (P(X1,Y1,Z1)) in the spatial coordinate system (SCS) to a first camera position (R1) in the camera coordinate system (CCS) based on an external conversion parameter determined by at least one of the position of the camera (300) in the spatial coordinate system and the shooting direction (or angle) of the camera (300).

[0079] For example, the processor (240) can convert a first camera position (R1) in the camera coordinate system (CCS) to a first image position (Q1(x1,y1)) in the image coordinate system (ICS) based on an internal transformation parameter determined by at least one of the focal length of the camera (300), the principal point of the camera (300), and the asymmetry coefficient of the camera (300).

[0080] According to embodiments of the present invention, the processor (240) can convert the spatial positions of speakers (SPK1 to SPK4) into image positions on an image (IMG).

[0081] Referring to FIG. 6, the processor (240) can generate output image data for displaying text associated with the voice of speakers (SPK1~SPK4) on an image (IMG) corresponding to the input image data using input image data. According to embodiments, the processor (240) can generate output image data by inserting text associated with the voice of speakers (SPK1~SPK4) at an image position corresponding to the image position data into an input image captured by a camera (300).

[0082] According to embodiments of the present invention, the processor (240) can generate output image data by inserting text associated with the voice of speakers (SPK1 to SPK4) into an input image captured by a camera (300) at the location of the speakers (SPK1 to SPK4) within the image (IMG) (i.e., image location). Accordingly, the user can easily identify who the speaker is that uttered the voice corresponding to the text by looking at the location of the text displayed in the image.

[0083] Additionally, for example, if at least a portion of the second speaker (SPK2) is located on the right side of the video display area, the processor (240) can generate output video data by inserting text to identify the second speaker (SPK2) into the input video along with the second text (TXT2) associated with the second voice "☆☆☆". If the second speaker (SPK2) is not displayed in the video, since the identity of the second speaker (SPK2) cannot be identified from the display position of the second text (TXT2), text to identify the second speaker (SPK2) (e.g., a name) is displayed together with the second text (TXT2).

[0084] The processor (240) can generate output image data by inserting a first text (TXT1) associated with the first voice "◎◎◎" of the first speaker (SPK1) at a position corresponding to the first image position (Q1) of the first speaker (SPK1) of the input image (IMG) corresponding to the input image data, and inserting a second text (TXT2) associated with the second voice "☆☆☆" of the second speaker (SPK2) at a position corresponding to the second image position (Q2) of the second speaker (SPK2).

[0085] For example, the processor (240) can convert the first output voice data of the first speaker (SPK1) into text data and, using the text data, insert a first text (TXT1) associated with the first voice "◎◎◎" of the first speaker (SPK1) at a position corresponding to the first image position (Q1) of the first speaker (SPK1) in the image (IMG) of the input image data. For example, the processor (240) can insert the first text (TXT1) at a position having a certain offset from the first image position corresponding to the first image position data.

[0086] The processor (240) can generate first setting data to determine the size, location, shape, and form of the area where the first text (TXT1) associated with the first voice "◎◎◎" is to be displayed, based on first image location data representing the first image location (Q1).

[0087] The processor (240) can store the generated output image data, and the image data output circuit (250) can transmit the output image data to the display device (400) under the control of the processor (240).

[0088] The display device (400) can display an image using output image data, and in the image displayed by the output image data, a first text (TXT1) associated with the first voice "◎◎◎" of the first speaker (SPK1) is displayed at a position corresponding to the first image position (Q1) of the first speaker (SPK1), and a second text (TXT2) associated with the second voice "☆☆☆" of the second speaker (SPK2) is displayed at a position corresponding to the second image position (Q2) of the second speaker (SPK2).

[0089] According to embodiments, the processor (240) can generate output image data so that text (TXT1 and TXT2) is displayed on the display device (400) in the form of a speech bubble. For example, the processor (240) can determine the size of the speech bubble, the position where the speech bubble is to be displayed, the direction of the speech bubble, etc., based on image positions (Q1 and Q2) within the image (IMG) of each speaker (SPK1 and SPK2).

[0090] An electronic device (200) according to embodiments of the present invention can determine the spatial location of each speaker (SPK1~SPK4) using input voice data associated with the voice of each speaker (SPK1~SPK4), determine the image location within the image based on the spatial location, and display text associated with the voice of each speaker (SPK1~SPK4) at a location corresponding to the image location within the image of the speakers (SPK1~SPK4).

[0091] FIG. 7 is a diagram illustrating the operation of an electronic device according to embodiments of the present invention. Referring to FIGS. 1 to 7, the electronic device (200) can generate output image data by inserting text (TXT1 and TXT2) associated with the voice of the speaker (SPK1 to SPK4) into the input image, even if the image position of the speaker (SPK1 to SPK4) is located outside the area where the image (IMG) is displayed (hereinafter referred to as the “image display area”).

[0092] Referring to FIG. 7, the second speaker (SPK2) is not displayed in the image of the input image data. Nevertheless, the electronic device (200) can generate output image data by inserting a second text (TXT2) associated with the second voice "☆☆☆" of the second speaker (SPK2) into the input image.

[0093] The processor (240) can determine whether at least a portion of the second speaker (SPK2) is displayed outside the image display area (e.g., to the right) by using input image data and the second image position (Q2) of the second speaker (SPK2). According to embodiments, the processor (240) can determine whether the second image position (Q2) exists outside the image display area (e.g., to the right) by using input image data and second image position data representing the second image position (Q2) of the second speaker (SPK2).

[0094] When at least a portion of the second speaker (SPK2) is displayed outside the video display area (e.g., to the right), the processor (240) can generate output video data by inserting a second text (TXT2) associated with the second voice "☆☆☆" of the second speaker (SPK2) into the input video. For example, the processor (240) can insert the second text (TXT2) associated with the second voice "☆☆☆" into a location adjacent to the second video position (Q2) within the video display area. For example, when at least a portion of the second speaker (SPK2) is located to the right of the video display area, the processor (240) can insert the second text (TXT2) into the right side of the video display area.

[0095] Additionally, for example, if at least a portion of the second speaker (SPK2) is located on the right side of the video display area, the processor (240) can generate output video data by inserting text to identify the second speaker (SPK2) into the input video along with the second text (TXT2) associated with the second voice "☆☆☆". If the second speaker (SPK2) is not displayed in the video, since the identity of the second speaker (SPK2) cannot be identified from the display position of the second text (TXT2), text to identify the second speaker (SPK2) (e.g., a name) is displayed together with the second text (TXT2).

[0096] As shown in FIG. 7, the second text (TXT) associated with the second voice "☆☆☆" can be inserted into the video along with the phrase "SPK2" representing the second speaker (SPK2).

[0097] When the first image position (Q1) of the first speaker (SPK1) is located inside the image display area and the second image position (Q2) of the second speaker (SPK2) is located outside the image display area, the processor (240) can insert a first text (TXT1) associated with the first voice "◎◎◎" of the first speaker (SPK1) into the input image as a first display type, and insert a second text (TXT2) associated with the second voice "☆☆☆" of the second speaker (SPK2) as a second display type. Depending on the embodiments, the first display type and the second display type may be different. For example, as shown in FIG. 7, the first text (TXT1) may be displayed in a solid line speech bubble, while the second text (TXT2) may be displayed in a dashed line speech bubble.

[0098] FIG. 8 is a flowchart for explaining the operation of an electronic device according to embodiments of the present invention. The operation method described with reference to FIG. 8 may be implemented as a program that can be stored on a computer-readable storage medium. Referring to FIGS. 1 through 8, the electronic device (200) may receive input voice data associated with the voice of a speaker (SPK1 to SPK4) (S110). According to embodiments, the electronic device (200) may receive input voice data using voice signals received from a plurality of microphones (100).

[0099] The electronic device (200) can generate spatial location data indicating the location of speakers (SPK1~SPK4) in space using input voice data (S120). According to embodiments, the electronic device (200) can generate spatial location data indicating the location of the sound source of the voice, i.e., the location of the speaker, based on input voice data.

[0100] The electronic device (200) can generate text data corresponding to the speaker's voice using input voice data (S130). According to embodiments, the electronic device (200) can generate output voice data associated with a voice spoken at a location corresponding to spatial location data using input voice data, and convert the output voice data into text data. For example, if the first spatial location data represents a first location (P1), the electronic device (200) can generate first output voice data associated with a voice spoken at the first location (P1) from the input voice data, and generate first text data using the first output voice data. At this time, the first text data may be data for representing the first voice of the first speaker “◎◎◎” as characters.

[0101] The electronic device (200) can convert spatial location data into image location data (S140). According to embodiments, the electronic device (200) can convert spatial location data representing the location of speakers (SPK1~SPK4) in a spatial coordinate system into image location data representing the location of speakers (SPK1~SPK4) in an image coordinate system using stored conversion parameters.

[0102] The electronic device (200) can generate output image data by inserting text associated with output voice data into an image position corresponding to the image position data of the input image (S150). According to embodiments, the electronic device (200) can generate output image data by synthesizing text associated with the input image data and the output image data according to the image position.

[0103] Although the embodiments have been described above with reference to limited embodiments and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0104] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0105] 100: Microphones 200: Electronic devices 300: Camera 400: Display device 210: Video data receiving circuit 220: Audio data receiving circuit 230: Memory 240: Processor 250: Video data output circuit

Claims

Claim 1 A video data receiving circuit configured to receive input video data associated with an input video captured by said camera from a camera; a voice data receiving circuit configured to receive input voice data associated with the voices of speakers; a memory configured to store transformation parameters for projecting a spatial coordinate system onto a video coordinate system on said input video; and a processor configured to generate output video data using said input voice data and input video data. An electronic device comprising a video data output circuit configured to output the output video data, wherein the processor generates, using the input voice data, first spatial location data representing a first spatial location in the spatial coordinate system of a first speaker among the speakers and first text data associated with the voice of the first speaker, converts the first spatial location data into first image location data representing a first image location in the image coordinate system of the first speaker based on the conversion parameter, determines whether the first spatial location is located within an image display area where the input video is displayed, and if the first spatial location is located inside the image display area, inserts a first text associated with the voice of the first speaker into the input video as a first display type according to the first image location, and if the first spatial location is located outside the image display area, generates the output video data by inserting a first text associated with the voice of the first speaker into the input video as a second display type different from the first display type to indicate that the first spatial location is located outside the image display area. Claim 2 An electronic device according to claim 1, wherein the input voice data is generated from voice signals generated by a plurality of microphones. Claim 3 In paragraph 2, the electronic device, wherein the processor calculates the first spatial position based on the distance between the plurality of microphones and the time at which the voice signal is received by the plurality of microphones. Claim 4 An electronic device according to claim 1, wherein the transformation parameter is a parameter determined based on an external transformation parameter for transforming the spatial coordinate system into a camera coordinate system on the camera; and an internal transformation parameter for transforming the camera coordinate system into an image coordinate system. Claim 5 An electronic device according to claim 4, wherein the external transformation parameter is determined based on at least one of the position of the camera in the spatial coordinate system and the shooting direction of the camera, and the internal transformation parameter is determined based on at least one of the focal length of the camera, the principal point of the camera, and the asymmetry coefficient of the camera. Claim 6 An electronic device according to claim 1, wherein the processor generates first setting data for determining the location, size, and shape of an area to be displayed of the first text using the first image position data, and generates the output image data by inserting the first text into the input image according to the first setting data. Claim 7 An electronic device according to claim 1, wherein the processor generates, using the input voice data, second spatial position data representing a second spatial position of a second speaker among the speakers in the spatial coordinate system and second text data associated with the voice of the second speaker, converts the second spatial position data into second image position data representing a second image position of the second speaker in the image coordinate system based on the conversion parameter, and generates the output image data by additionally inserting second text associated with the voice of the second speaker into the input image according to the second image position based on the input image data, the second image position data, and the second text data. Claim 8 delete Claim 9 A method of operating an electronic device comprises: receiving input image data associated with an input image captured by the camera from a camera; receiving input voice data associated with the voices of speakers; using the input voice data, generating first spatial location data representing a first spatial location in a spatial coordinate system of a first speaker among the speakers and first text data associated with the voice of the first speaker; converting the first spatial location data into first image location data representing a first image location in the image coordinate system of the input image of the first speaker; determining whether the first spatial location is located within an image display area where the input image is displayed; if the first spatial location is located inside the image display area, inserting first text associated with the voice of the first speaker according to the first image location into the input image as a first display type, and if the first spatial location is located outside the image display area, inserting first text associated with the voice of the first speaker according to the first image location into the input image as a first display type different from the first display type to indicate that the first spatial location is located outside the image display area. A method of operating an electronic device comprising: a step of generating output image data by inserting it into a second display type; and a step of outputting the output image data. Claim 10 A method of operation of an electronic device according to claim 9, wherein the step of converting the first spatial position data into the first image position data comprises the step of converting the first spatial position data into the first image position data using a transformation parameter for projecting the spatial coordinate system onto the image coordinate system. Claim 11 In claim 9, the method of operating the electronic device further comprises: a step of generating, using the input voice data, second spatial position data representing a second spatial position of a second speaker among the speakers in the spatial coordinate system and second text data associated with the voice of the second speaker; a step of converting the second spatial position data into second image position data representing a second image position of the second speaker in the image coordinate system; and a step of generating the output image data by additionally inserting second text associated with the voice of the second speaker into the input image according to the second image position, based on the input image data, the second image position data, and the second text data. Claim 12 delete

Images