User terminal, video calling device, video calling system and its control method

Abstract

A user terminal, a video call translation device, a video call translation system including the device, and a control method thereof are disclosed. According to one aspect, the video call translation device may include: a communication unit supporting video call services between multiple user terminals via a communication network; an extraction unit generating image files and audio files using video call-related video files collected from each of the multiple user terminals, and extracting original language information from at least one of the image files and audio files; a translation unit generating translation information from the original language information; and a control unit for controlling the transmission of the interpretation or translation video, wherein the interpretation or translation video is formed by mapping at least one of the extracted original language information and translation information to the video call-related video files.

Description

Technical Field

[0001] This invention relates to user terminals, video calling devices, video calling systems, and control methods thereof, and is applicable not only to one-to-one video calls, but also to providing real-time original text / translation services during video calls between multiple people. Background Technology

[0002] With the development of IT technology, video calls between users are becoming more frequent, especially among people around the world who use video call services not only for business purposes, but also for content sharing and sharing hobbies and lifestyles.

[0003] However, requiring an interpreter for each video call presents challenges in terms of cost and time, so research is underway on how to provide real-time original text / translation services for video calls. Summary of the Invention

[0004] The technical problem that the invention aims to solve

[0005] The purpose of this invention is to facilitate smoother communication and understanding of needs by providing real-time original text / translation services between speakers of various languages. Furthermore, by providing original text / translation services through at least one of voice and text, visually and hearing impaired individuals can also communicate freely and understand each other's needs smoothly. The invention also provides various functions to facilitate communication, such as electronic whiteboard, text transmission, and speaking rights settings.

[0006] Technical solutions for solving the problem

[0007] A video call device according to one aspect may include: a communication unit that supports video call services between multiple user terminals via a communication network; an extraction unit that generates image files and audio files using video call-related video files collected from each of the multiple user terminals, and extracts original language information from at least one of the image files and audio files; a translation unit that generates translation information from the original language information; and a control unit for controlling the transmission of interpreting or translated video, wherein the interpreting or translated video is formed by mapping at least one of the extracted original language information and translation information to the video call-related video.

[0008] Furthermore, the original language information may include at least one of speech original language information and text original language information, and the translation information may include at least one of speech translation information and text translation information.

[0009] Furthermore, the extraction unit can perform a frequency band analysis process on the voice file to extract the original speech information of each speaker, and perform a speech recognition process on the extracted original speech information to generate original text information.

[0010] Furthermore, the extraction unit can perform image processing on the image file to detect sign language patterns, and extract the original language information of the text based on the detected sign language patterns.

[0011] According to one aspect, the user terminal may include: a terminal communication unit that supports video call services via a communication network; and a terminal control unit configured to provide interpretation or translation video, wherein the interpretation or translation video is generated by mapping at least one of the original language information and translation information to a video call-related video file, and a control display shows a user interface configured to provide icons for receiving at least one or more video call-related setting instructions and at least one or more translation-related setting instructions.

[0012] Furthermore, the at least one or more video call related setting instructions may include at least one of the following: a speaking rights setting instruction that can set the speaking rights of video call participants, a video call participant number setting instruction, a blackboard activation instruction, and a text transmission instruction.

[0013] Furthermore, the terminal control unit can control the display to show a user interface configured to change the control method of the interpretation or translation video or provide a pop-up message including information about the person with the right to speak, depending on whether the speaking right setting instruction is input.

[0014] Furthermore, the terminal control unit can control the display to show a user interface configured to provide a virtual keyboard in a preset area when the text transmission command is received.

[0015] A video call device according to one aspect may include: receiving video call-related video files from multiple user terminals via a communication network; extracting original language information of each caller using at least one of image files and audio files generated from the video call-related video files; generating translation information that translates the original language information according to a selected national language; and controlling the transmission of interpreting or translated video, wherein the interpreting or translated video is generated by mapping at least one of the original language information and the translation information to the video call-related video files.

[0016] Furthermore, the extraction steps may include: performing a frequency band analysis process on the audio file to extract the original speech language information of each speaker; and performing a speech recognition process on the extracted original speech language information to generate original text language information.

[0017] The effects of the invention

[0018] According to one embodiment, a user terminal, a video calling device, a video calling system including the device, and a control method thereof facilitate smoother communication and understanding by providing real-time original / translation services between callers using various languages.

[0019] According to another embodiment, a user terminal, a video calling device, a video calling system including the device, and a control method thereof provide original / translation services via at least one of voice and text, enabling visually and hearing impaired individuals to communicate freely and understand each other's needs smoothly.

[0020] According to one embodiment, the user terminal, video calling device, video calling system including the device, and control method thereof also provide various functions for facilitating communication, such as electronic whiteboard function, text transmission function, speaking rights setting function, etc., so as to conduct more effective video calls. Attached Figure Description

[0021] Figure 1 The accompanying drawings are for illustrating various user terminals according to one embodiment.

[0022] Figure 2 The accompanying drawings are for illustrative purposes only and are intended to briefly show the structure of a video call system according to one embodiment.

[0023] Figure 3 The accompanying drawings are provided to briefly illustrate the user interface screen displayed on a monitor during a video call between two people according to one embodiment.

[0024] Figure 4 This diagram briefly illustrates the user interface screen displayed on the monitor during a video call between five participants according to one embodiment.

[0025] Figure 5 The accompanying drawings are for illustrative purposes only and show the user interface screen displayed on the monitor when one of the five people in a call has the right to speak, according to one embodiment.

[0026] Figure 6 The accompanying drawings illustrate a user interface screen configured to receive various setting instructions according to one embodiment.

[0027] Figure 7 The accompanying drawings are for illustrative purposes only and illustrate the operation flowchart of a video call device according to one embodiment. Detailed Implementation

[0028] Figure 1 The accompanying drawings are for illustrating various user terminals according to one embodiment. Figure 2 The accompanying drawings are provided to briefly illustrate the structure of a video call system according to one embodiment. Figure 3The accompanying drawings are provided to briefly illustrate the user interface screen displayed on a monitor during a video call between two people according to one embodiment. Figure 4 The accompanying drawings are provided to briefly illustrate the user interface screen displayed on the monitor during a video call between five participants according to one embodiment. Figure 5 The accompanying drawings are provided to briefly illustrate the user interface screen displayed on the monitor when one of the five conversation participants has the right to speak, according to one embodiment. Figure 6 The accompanying drawings illustrate a user interface screen configured to receive various setting instructions according to one embodiment. These will be described in detail below to avoid repetition.

[0029] The user terminals described below include all devices that have a built-in processor capable of handling various calculations and a built-in display and speakers to support video call services for users. For example, user terminals include such Figure 1 The desktop computer S1 and tablet computer S2 shown are examples of this, and in addition, portable terminals such as… Figure 1 The smartphone S3 shown includes a wearable device S4 in the form of a watch or glasses that can be detached from the user's body, as well as other related devices. Figure 1 The TV S5 shown includes smart TVs, IPTV, etc., but is not limited to these.

[0030] For ease of explanation, the following description uses a smartphone as an example among the various types of user terminals mentioned above; however, the invention is not limited to this. For ease of explanation, the person using the user terminal to access the video call service will be referred to as the user or the caller; these two terms can be used interchangeably.

[0031] On the other hand, the video calling device described below includes all devices that have a built-in communication module capable of sending and receiving various types of data through a communication network and a processor capable of performing various computational processes. For example, video calling devices may include the aforementioned laptops, desktop computers, tablets, smartphones, PDAs, wearable terminals, and may also include smart TVs, IPTV, etc., as well as servers with built-in communication modules and processors, but are not limited thereto.

[0032] Reference Figure 2 The video call system 1 supports video calls between user terminals 200 (200-1, ..., 200-n) (n≥1) and user terminals 200, including a video call device 100 for providing original / translation services for video calls.

[0033] Reference Figure 2The video call device 100 may include: a communication unit 110 that supports video call services between user terminals 200 via a communication network; an extraction unit 120 that generates image files and audio files using video call-related video files received through the communication unit 110, and then extracts the original language information based on these; a translation unit 130 that translates the original language information to generate translation information; and a control unit 140 that controls the overall operation of the structural elements in the video call device 100 to provide translation information.

[0034] The communication unit 110, extraction unit 120, translation unit 130, and control unit 140 can be implemented individually, or at least one can be integrated into a single system-on-a-chip (SOC). However, since the video call device 100 may contain more than one SOC, it is not limited to integration into a single SOC, and the implementation is not limited. The structural elements of the video call device 100 will be described in detail below.

[0035] The communications unit 110 can exchange various data with external devices through a wireless communication network or a wired communication network. The wireless communication network refers to a communication network capable of wirelessly sending and receiving signals, including data.

[0036] For example, the communication unit 110 transmits and receives wireless signals between devices via base stations using communication methods such as 3G (3 Generation), 4G (4 Generation), and 5G (5 Generation). Furthermore, it can transmit and receive wireless signals, including data, with terminals within a predetermined distance using communication methods such as Wireless LAN, Wi-Fi, Bluetooth, Zigbee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), and Near Field Communication (NFC).

[0037] Furthermore, a wired communication network refers to a communication network that can send and receive signals, including data, via a wired connection. Examples of wired communication networks include Peripheral Component Interconnect (PCI), PCI-express, and Universal Serial Bus (USB), but are not limited to these. The communication networks described below include both wireless and wired communication networks.

[0038] The communication unit 110 can receive video call-related video files from the user terminal 200 during a video call via video call service. The video call-related video files are data received from the user terminal 200 during a video call and may include image information providing visual information and voice information providing auditory information.

[0039] The control unit 140 controls the communication unit 110 to support video calls based on the request from the user terminal 200. It can send only video call-related video files, or it can send an interpretation or translation video file mapped from at least one of the original language information and translation information to the video call-related video file. In addition, it can transmit various files necessary for communication between the callers, such as image files created via the electronic whiteboard function or text files created via the text function. For a detailed description of the control unit 140, please refer to the following description.

[0040] Reference Figure 2 The video call device 100 may include an extraction unit 120. The extraction unit 120 may generate image files and audio files using video call-related video files received through the communication unit 110.

[0041] Image files and audio files contain language information, and the extraction unit 120 according to the embodiment can extract the original language information from the image files and audio files. The original language information described below is information extracted from communication means such as voice and sign language included in the video, and the original language information can be extracted from voice or text.

[0042] For ease of explanation, the raw language information consisting of speech will be referred to as raw speech language information, and the raw language information consisting of text will be referred to as raw text language information. For example, when a person (the person making the call) in a video call says "Hello" in English, the raw speech language information is the spoken "Hello" by the person making the call, and the raw text language information is the text "Hello" itself. The following section first describes the method for extracting raw speech language information from audio files.

[0043] Image files may contain a mix of voices from various speakers. Providing such a variety of voices simultaneously can confuse users and make translation difficult. Therefore, the extraction unit 120 can extract the original speech information of each speaker from the voice file through a frequency band analysis process.

[0044] Depending on factors such as gender, age group, tone of voice, and intensity of speech, each person's voice may differ. By analyzing frequency bands, it is possible to distinguish the speakers. Therefore, the extraction unit 120 can extract the original speech language information by analyzing the frequency bands of the speech file and separating the speech according to each person appearing in the video based on the analysis results.

[0045] The extraction unit 120 can generate original text language information that converts the original speech language information into text, and then distinguish and store the original speech language information and original text language information according to different callers.

[0046] The methods for analyzing the frequency bands of voice files and converting raw voice language information into raw text language information can be implemented as data in the form of algorithms or programs and pre-stored in the video call device 100. The extraction unit 120 can use the pre-stored data to separate and generate raw language information.

[0047] On the other hand, during video calls, certain participants may use sign language. In this case, unlike the aforementioned method of extracting raw speech information from an audio file and then generating raw text information from that raw speech information, the extraction unit 120 can directly extract raw text information from an image file. The method for extracting raw text information from an image file will be described below.

[0048] The extraction unit 120 can perform an image processing procedure on the image file to detect sign language patterns and generate original text language information based on the detected sign language patterns.

[0049] Whether to perform an image processing procedure can be set automatically or manually. For example, when a sign language translation request instruction is received from the user terminal 200 via the communication unit 110, the extraction unit 120 can detect the sign language pattern through the image processing procedure. As another example, the extraction unit 120 can automatically perform an image processing procedure on the image file to determine whether a sign language pattern exists in the image file, etc., and the present invention does not limit this.

[0050] The method for detecting sign language patterns through image processing can be implemented as data in the form of algorithms or programs and pre-stored in the video call device 100. The extraction unit 120 can use the pre-stored data to detect sign language patterns included in the image file and generate original text language information based on the detected sign language patterns.

[0051] The extraction unit 120 can map and store the original language information with specific person information.

[0052] For example, after the extraction unit 120 identifies the user terminal 200 that sent a specific voice, it maps the preset ID of the user terminal 200 or the preset nickname of the user (caller) to the original language information. Even if multiple users speak at the same time, the audience can accurately know which caller said what.

[0053] As another example, when a video call-related video file includes multiple callers, the extraction unit 120 can adaptively set the person information according to a preset method or the characteristics of the callers detected from the video call-related video file. In one embodiment, the extraction unit 120 can obtain the gender, age group, etc. of the person speaking through a frequency band analysis process, and arbitrarily set the name of the most suitable person based on the obtained results for mapping.

[0054] The control unit 140 can control the communication unit 110 to send raw language information and translated information mapped to the speaker information to the user terminal 200, thus making it easier for the user to identify who the speaker is. For a detailed description of the control unit 140, please refer to the following description.

[0055] Reference Figure 2 The video call device 100 may include a translation unit 130. The translation unit 130 can translate the original language information into the language required by the caller to generate translated information. When generating translated information according to the language input by the caller, the translation unit 130 can generate a translation result as text or as speech. The video call system 1 according to the embodiment provides the original language information and the translated information as speech or text respectively, thus having the advantage that both hearing-impaired and visually-impaired individuals can utilize the video call service.

[0056] For ease of explanation, the translation of original language information into the user's desired language will be referred to as translated information. Translated information, like the original language information, is also presented in either audio or text form. In this context, translated information presented as text will be called text-based translated information, and translated information presented as audio will be called audio-based translated information.

[0057] The voice translation information is voice information dubbed with a specific voice. The translation unit 130 can generate voice translation information dubbed with a preset voice or a user-set tone. Each user may want to hear a different tone. For example, one user may want voice translation information with a male tone, while another user may want voice translation information with a female tone. Therefore, the translation unit 130 can generate voice translation information with various tones so that users can watch more comfortably. Alternatively, the translation unit 130 can generate voice translation information with a tone similar to the speaker's voice based on the analysis of the speaker's voice; the present invention does not limit this. The video call device 100 according to the embodiment makes it easier for visually impaired people to receive video call services by providing voice translation information.

[0058] The translation method and the method of setting the tone of voice used during translation can be implemented as data in the form of algorithms or programs and pre-stored in the video call device 100. The translation unit 130 can use the pre-stored data to perform translation.

[0059] Reference Figure 2 The video call device 100 may include a control unit 140 for controlling the overall operation of the structural elements in the video call device 100.

[0060] The control unit 140 may be implemented by a processor capable of processing various calculations, such as a microcontroller unit (MCU), and a memory used to store control programs or control data for controlling the operation of the video call device 100, or to temporarily store control instruction data or image data output by the processor.

[0061] At this point, the processor and memory can be integrated into the system-on-a-chip (SOC) built into the video calling device 100. However, the video calling device 100 may have more than one built-in SOC, so it is not limited to integration into a single SOC.

[0062] The memory may include volatile memory (also known as temporary storage memory) such as SRAM and DRAM, flash memory, read-only memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and other non-volatile memory. However, the invention is not limited thereto and can be implemented in any other form known in the art.

[0063] In one embodiment, the non-volatile memory may store control programs and control data for controlling the operation of the video call device 100. The volatile memory may import control programs and control data from the non-volatile memory and store them temporarily, or temporarily store control instruction data output by the processor, etc. The present invention does not limit this.

[0064] The control unit 140 can generate control signals based on the data stored in the memory, and control the overall operation of the structural elements in the video call device 100 by generating control signals.

[0065] For example, the control unit 140 can control the communication unit 110 via control signals to support video calls. Furthermore, the control unit 140 can control the extraction unit 120 via control signals to generate image and audio files from video call-related files, such as video call-related video files, and extract original language information from at least one of the image and audio files.

[0066] The control unit 140 can generate and transmit interpreting or translation videos according to user terminals to facilitate smooth communication between users in various countries. The interpreting or translation videos are generated by mapping at least one of the original language information and the translation information to video call-related video files received from multiple user terminals.

[0067] At this point, the interpretation or translation video may only map the original language information or the translation information, or it may map both the original language information and the translation information simultaneously.

[0068] For example, when the interpretation or translation video only maps the original language information and the translated text, the interpretation or translation video can include the original language information and the translated text information related to each speech in the speech video as subtitles. As another example, when the interpretation or translation video maps both speech translation information and text translation information, the interpretation or translation video can include speech translation information translated into a specific language in the form of dubbing in the form of voiceover, and include the text translation information as subtitles in the form of text translations.

[0069] On the other hand, the control unit 140 can change the method of providing video call service and original / translation service based on the setting instructions or preset methods received from the user terminal 200 through the communication unit 110.

[0070] For example, when the communication unit 110 receives a video call participant number setting instruction from the user terminal 200, the control unit 140 can restrict the user terminal 200's access in accordance with the instruction.

[0071] As another example, when receiving individual text or image data from user terminal 200 via communication unit 110, control unit 140 can transmit the received text or image data together with the interpretation or translation video file, thereby making the exchange of opinions between the callers more accurate.

[0072] As another example, when the communication unit 110 receives a speaking rights setting instruction from the user terminal 200, such as a speaking restriction instruction or a speaking order related instruction, the control unit 140 may, in accordance with the instruction, only transmit the interpretation or translation video of the user terminal with speaking rights among the multiple user terminals 200. Alternatively, the control unit 140 may, in accordance with the instruction, transmit a pop-up message including content about speaking rights along with the interpretation or translation video. The present invention does not limit the implementation method.

[0073] As described below, the user terminal 200 may pre-store a program that supports video call services and translation services, and makes various settings according to each user's preferences to support the aforementioned services. Users can use this program to make various settings. The user terminal 200 will be described below.

[0074] Reference Figure 2 The user terminal 200 may include: a display 210 (210-1, ..., 210-n) for providing various information to the user visually; a speaker 220 (220-1, ..., 220-n) for providing various information to the user auditorily; a terminal communication unit 230 (230-1, ..., 230-n) for exchanging various data with external devices through a communication network; and a terminal control unit 240 (240-1, ..., 240-n) for controlling the overall operation of the structural elements in the user terminal 100 to support video call services (n≥1).

[0075] The terminal communication unit 230 and the terminal control unit 240 can be implemented separately or integrated into a system-on-a-chip (SOC), and the present invention does not impose any restrictions on this. The various structural elements of the user terminal 100 will be described below.

[0076] User terminal 200 may include a display 210 for providing various information to the user visually. According to one embodiment, display 210 may be implemented as a liquid crystal display (LCD), a light-emitting diode (LED), a plasma display panel (PDP), an organic light-emitting diode (OLED), a cathode ray tube (CRT), etc., but is not limited thereto. On the other hand, when display 210 is implemented as a touch screen panel (TSP), the user can touch a specific area of ​​display 210 to input various interpretive commands.

[0077] The display 210 can not only display video related to video calls, but also receive various control commands through the user interface displayed on the display 210.

[0078] The user interface described below may be a graphical user interface, which displays the screen on the monitor 210 in graphical form, thereby facilitating the exchange of various information and commands between the user and the user terminal 200.

[0079] For example, in a graphical user interface, a portion of the screen displayed on the display 210 shows icons, buttons, etc., for easy access to various control commands from the user, while other portions of the screen display various information through at least one widget. This invention does not limit this.

[0080] For example, such as Figure 3 As shown, the display 210 shows the video of the person making the video call and the other party, and can also display an icon I1 for inputting translation instructions, an icon I2 for receiving various setting instructions, an emoticon I3 for providing information on the video call service status, and a graphical user interface for providing original language / translation information M.

[0081] The terminal control unit 240 can control the display 210 to show, for example, the content displayed on the display 210 via control signals. Figure 3 The graphical user interface shown is described below. The display methods and layout methods of widgets, icons, emoticons, etc., that constitute the user interface can be implemented as data in the form of algorithms or programs, and are pre-stored in the memory of the user terminal 200 or the memory of the video call device 100. Therefore, the terminal control unit 240 uses the pre-stored data to generate control signals, and controls the display of the graphical user interface by means of the generated control signals. For a detailed description of the terminal control unit 240, please refer to the following description.

[0082] On the other hand, refer to Figure 2 The user terminal 200 may include a speaker 220 for outputting various sounds. The speaker 220 is disposed on one side of the user terminal 200 and can output various sounds included in video files related to video calls; the types of sounds that can be output are not limited. The speaker 220 can be implemented using various known sound output devices, and there are no limitations.

[0083] User terminal 200 may include terminal communication unit 230, which exchanges various data with external devices through a communication network.

[0084] The terminal communication unit 230 can exchange various data with external devices through a wireless communication network or a wired communication network. For a detailed description of the wireless communication network or the wired communication network, please refer to the foregoing content; therefore, a detailed description is omitted here.

[0085] The terminal communication unit 230 can exchange video call-related video files, interpretation or translation video files, etc. with other user terminals through the video call device 100 to provide video call services.

[0086] Reference Figure 2 The user terminal 200 may include a terminal control unit 240 for controlling the overall operation of the user terminal 200.

[0087] The terminal control unit 240 may be implemented by a processor such as a microcontroller unit (MCU) capable of processing various calculations, and a memory used to store control programs or control data that control the operation of the user terminal 200, or to temporarily store control instruction data or image data output by the processor.

[0088] At this point, the processor and memory can be integrated into the system-on-a-chip (SoC) built into the user terminal 200. However, the user terminal 200 may have more than one SoC built into it, so it is not limited to integration into a single SoC.

[0089] The memory may include volatile memory (also known as temporary storage memory) such as SRAM and DRAM, flash memory, read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, and other non-volatile memory. However, the invention is not limited thereto and can be implemented in any other form known in the art.

[0090] In one embodiment, the non-volatile memory may store control programs and control data for controlling the operation of the user terminal 200. The volatile memory may import control programs and control data from the non-volatile memory and store them temporarily, or temporarily store control instruction data output by the processor, etc. The present invention does not limit this.

[0091] The terminal control unit 240 can generate control signals based on the data stored in the memory, and control the overall operation of the structural elements in the user terminal 200 through the generated control signals.

[0092] For example, the terminal control unit 240 can control the display 210 to show various information via control signals. In the case of receiving a video caller's interpretation or translation video from the video call device 100 via the terminal communication unit 230, such as... Figure 3 As shown, the terminal control unit 240 can display the interpreting or translation video of the other party in a video call on the display 210.

[0093] Furthermore, the terminal control unit 240 can control the display 210 to show a user interface capable of receiving various setting instructions for video call services, and change the configuration of the user interface based on the setting instructions received through the user interface.

[0094] For example, when a user clicks on such Figure 3 When icon I2 is shown, the terminal control unit 240 can control the reduction of the area used to display video call-related interpretation or translation video to, for example, [the area is shown in the image]. Figure 4 As shown, a user interface configured to display icons that can receive various setting instructions from the user is shown on the display 210.

[0095] Specifically, refer to Figure 4The terminal control unit 240 can control the display 210 to display the following user interface, which includes icons for receiving video call participant invitation instructions, translation language selection instructions, speaking rights setting instructions, electronic whiteboard instructions, keyboard activation instructions, subtitle setting instructions, and other setting instructions. The setting instructions that can be entered are not limited to the examples mentioned above.

[0096] The video call system 1 according to the embodiment can provide not only one-to-one video calls, but also video call services between multiple people. Therefore, when a user clicks the video call invite icon to invite other users, the terminal control unit 240 can add and divide the area for displaying the video call-related video according to the number of invited users. In one embodiment, when a user invites two more people to a video call during a video call with one person, for a total of three people participating in the video call, such as... Figure 5 As shown, the terminal control unit 240 can display a user interface on the display 210 configured to show the videos of three callers in the first to third areas R1, R2, and R3 respectively, and to show the original language / translation information M1, M2, and M3 of each caller in the first to third areas R1, R2, and R3 respectively. If another caller is invited, the terminal control unit 240 can display the video and original language / translation information of the newly added caller in the fourth area R4; this invention does not limit this.

[0097] On the other hand, when a user clicks the speaking rights setting icon to make speaking rights related settings, the terminal control unit 240 can highlight the video of the user with speaking rights through various methods.

[0098] For example, such as Figure 6 As shown, the terminal control unit 240 can control the user interface displayed on the display 210 to magnify the video of the person with speaking rights and provide only the original language / translation information M1 of the user with speaking rights. As another example, the terminal control unit 240 can change the user interface displayed on the display 210 to provide only the video and original language / translation information of the person with speaking rights. The terminal control unit 240 can change the user interface in various ways to distinguish between people with speaking rights and those without speaking rights; this invention does not limit this.

[0099] The aforementioned user interface configuration method can be implemented as data in the form of algorithms or programs, and pre-stored in the user terminal 200 or the video call device 100. When pre-stored in the video call device 100, the terminal control unit 240 can receive the aforementioned data from the video call device 100 through the terminal communication unit 230, and then control the display of the user interface on the display 210 based on this data. The operation of the video call device will be briefly described below.

[0100] Figure 7 This is a flowchart illustrating the operation of a video call device according to one embodiment.

[0101] Video calling devices can provide video calling services by connecting multiple user terminals through a communication network. In this case, the user terminals can receive video files related to the video call. The video files related to the video call are data generated using at least one of the camera and microphone built into the user terminal, and can refer to data storing the user's communication content through at least one of the aforementioned camera and microphone.

[0102] The video call device can generate image files and audio files for each user terminal based on video call-related video files received from each user terminal (step 700), and extract the original language information of each user terminal using at least one of the generated image files and audio files (step 710).

[0103] The original language information refers to the information that represents the communication content stored in the video related to the video call in at least one form of speech and text, which is equivalent to the information before it is translated into a specific language.

[0104] Video calling devices can extract raw language information using all or one of image and audio files, based on the communication methods used by the callers in the relevant video.

[0105] For example, when one person in a video call uses voice while the other uses sign language, the video call device can identify the sign language pattern from the image file to extract the original language information, and identify the voice from the audio file to extract the original language information.

[0106] As another example, when multiple callers use only voice for video calls, the video call device can extract the original language information using only the voice file. As yet another example, when multiple callers use only sign language for conversation, the video call device can extract the original language information using only the image file.

[0107] The video calling device can generate translation information using the original language information based on the requests of multiple callers (step 720), and then provide at least one of the original language information and the translation information through a communication network (step 730). For example, the video calling device transmits an interpreting or translated video that maps at least one of the original language information and the translation information to the video call-related video, enabling smooth communication between the callers.

[0108] The embodiments described in the specification and the configurations shown in the accompanying drawings are merely preferred examples of the disclosed invention, and various modifications that can replace the embodiments and drawings in this specification may exist at the time of filing this application.

[0109] Furthermore, the terminology used in this specification is for describing embodiments and is not intended to limit and / or limit the disclosed invention. Unless the context clearly specifies otherwise, singular expressions include plural expressions. In this specification, terms such as "comprising" or "having" are intended to indicate the presence of the features, quantities, steps, operations, structural elements, components, or combinations thereof described in the specification, without precluding the presence or additional possibilities of one or more other features, quantities, steps, operations, structural elements, components, or combinations thereof.

[0110] Furthermore, the terms including ordinal numbers such as "first" and "second" used in this specification can be used to describe various structural elements, but the structural elements are not limited by these terms; the terms are only used to distinguish one structural element from another. For example, without departing from the scope of the invention, a first structural element can be referred to as a second structural element, and similarly, a second structural element can be referred to as a first structural element. The term "and / or" includes a combination of multiple related listed items or any one of multiple related listed items.

[0111] Furthermore, terms such as “unit,” “group,” “block,” “member,” and “module” used throughout this specification can refer to a unit used to perform at least one function or operation. For example, it may mean software or hardware, such as an FPGA or ASIC. However, “unit,” “group,” “block,” “member,” and “module” are not limited to software or hardware; they can be structural elements stored in accessible storage media and executed by one or more processors.

Claims

1. A video call device, characterized in that, include: The Ministry of Communications supports video call services between multiple user terminals through the communication network; The extraction unit uses video call-related video files collected from each of the plurality of user terminals to generate image files and audio files, and uses the image files and audio files to extract the original language information of each user, wherein the original language information includes original voice language information and original text language information. The translation department generates translation information from the original language information, wherein the translation information includes speech translation information and text translation information; and The control unit controls the transmission of interpretation or translation video, wherein the interpretation or translation video is generated by mapping the extracted original language information and translation information to the video call-related video. The extraction unit performs a frequency band analysis process on the voice file to extract the original speech information of each caller, and maps the extracted original speech information to specific person information for storage. The mapping involves the extraction unit identifying the user terminal that sent the specific voice message and mapping the user terminal's preset ID or the user's preset nickname to the original speech information. The extraction unit further performs a speech recognition process on the extracted raw speech information to generate raw text information, and performs an image processing process on the image file to determine whether a sign language pattern exists in the image file. If a sign language pattern exists, raw text information is generated based on the detected sign language pattern. The translation unit, based on the characteristics of each speech segment obtained from the frequency band analysis process of the speech file by the extraction unit, generates speech translation information using a preset set of speech segments that are most similar to the speaker's voice. These characteristics include the speaker's gender, age group, and pronunciation pitch. Upon receiving a speaking rights setting instruction from the user terminal, the control unit, in response to the speaking rights setting instruction, has the translation unit translate only the original language information collected from the user terminals that have set speaking rights among the multiple user terminals, and transmit the translated interpretation or translated video.

2. A control method for a video call device, characterized in that, include: Receive video files related to video calls from multiple user terminals via a communication network; Using image files and audio files generated from the video files related to the video call, the original language information of each caller is extracted, wherein the original language information includes original voice language information and original text language information; Generate translation information based on the selected country's language, translating the original language information, wherein the translation information includes speech translation information and text translation information; and The interpretation or translation video is transmitted by controlling the transmission of the video, wherein the interpretation or translation video is generated by mapping the original language information and the translation information to the video call-related video file. In the extraction of raw language information, The audio file undergoes a frequency band analysis process to extract the raw speech information of each speaker. This extracted raw speech information is then mapped and stored with specific speaker information. The mapping involves the extraction unit identifying the user terminal sending the specific speech and mapping the user terminal's preset ID or the user's preset nickname to the raw speech information. The extraction of raw language information includes: The extracted raw speech information is subjected to a speech recognition process to generate raw text information. The image file is subjected to an image processing process to determine whether a sign language pattern exists in the image file. If a sign language pattern exists, raw text information is generated based on the detected sign language pattern. The generated translation information includes: In the extraction step, based on the characteristics of each voice obtained through frequency band analysis of the voice file, speech translation information is generated using the voice most similar to the speaker's voice from a preset pool of voices. These characteristics include the speaker's gender, age group, and pronunciation pitch. The control of transmitting interpretation or translation video includes: Upon receiving a speaking rights setting instruction from the user terminal, in response to the speaking rights setting instruction, only the original language information collected from the user terminals that have set speaking rights among the multiple user terminals is translated, and the translated interpretation or translated video is transmitted.

Images