Displaying sign language videos through adjustable user interface (UI) elements

The electronic device enhances the visibility of sign language videos by determining signer location and displaying them in adjustable picture-in-picture windows, addressing the inconvenience of obstructive sign language displays in existing systems.

JP2024542064AInactive Publication Date: 2024-11-13SONY GROUP CORP
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2024525986
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-04
Filing Date
2022-10-21
Publication Date
2024-11-13
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing systems do not provide easy user interface techniques for viewers with hearing impairments to conveniently view media content and sign language videos, as sign language videos are often too small or obstructive to main video content.

Method used

An electronic device that determines the location of a signer in a video and extracts a portion of the video for display in a picture-in-picture window with adjustable size and position, using metadata or neural network models to detect sign language gestures, allowing customization based on user preferences.

Benefits of technology

Provides a clear and enlarged display of sign language videos without obstructing main video content, offering an unobstructed view of important notifications and customizable user interface elements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542064000001_ABST
    Figure 2024542064000001_ABST
Patent Text Reader

Abstract

An electronic device and method for display of sign language video through adjustable user interface (UI) elements. The electronic device receives a first media stream including a video. The electronic device determines a position of a signer in the video. The electronic device extracts a video portion that corresponds to the determined position in the video. The electronic device controls playback of the video on a display device. The electronic device controls the display device based on the playback to render UI elements on the display device and display the extracted video portion in the UI element.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [CROSS REFERENCE / INCORPORATION BY REFERENCE TO RELATED APPLICATIONS]

[0001] This application claims the benefit of priority to U.S. Patent Application No. 17 / 453,553, filed in the United States Patent Office on November 4, 2021. Each of the above applications is incorporated herein by reference in its entirety.

[0002]

[0002] Various embodiments of the present disclosure relate to displaying sign language videos. More specifically, various embodiments of the present disclosure relate to electronic devices and methods for displaying sign language videos through adjustable UI elements. [Background technology]

[0003]

[0003] Conventionally, a display device (such as a television or mobile phone) receives broadcast or streaming media content including a video file and an audio file synchronized with the video file. Both the video file and the audio file are rendered and viewed simultaneously on the display device. In some examples, the media content (e.g., a video) also includes a signer or interpreter who uses sign language (e.g., American Sign Language (ASL)) to interact with a hearing-impaired viewer. However, the video portion of the signer (sign language video) typically appears in a bottom corner of the video. The sign language video may be too small to be viewed comfortably or may obstruct areas of the main video (such as important notifications in the main video). Existing systems do not provide simple user interface techniques to assist hearing-impaired viewers in conveniently viewing the media content and the sign language video.

[0004]

[0004] Further limitations and disadvantages of conventional methods will become apparent to those skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remaining portions of this application and with reference to the drawings. Summary of the Invention [Problem to be solved by the invention]

[0005]

[0005] Provided is an electronic device and method for displaying sign language video through adjustable UI elements, as substantially shown in and / or described in connection with at least one figure and more fully set forth in the claims.

[0006]

[0006] These and other features and advantages of the present disclosure can be understood by considering the following detailed description of the disclosure in conjunction with the accompanying drawings, in which like elements are designated with like reference numerals throughout. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an exemplary network environment for displaying sign language video through adjustable user interface (UI) elements in accordance with an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of an exemplary electronic device for displaying sign language video through adjustable user interface (UI) elements, according to an embodiment of the present disclosure. [Diagram 3] A diagram illustrating an example scenario for displaying sign language video through adjustable user interface (UI) elements based on metadata, in accordance with an embodiment of the present disclosure. [Figure 4] A diagram illustrating an example scenario for displaying sign language video through adjustable user interface (UI) elements for live video broadcasting, according to an embodiment of the present disclosure. [Diagram 5] 1A-1C illustrate example scenarios for adjusting the position of a user interface (UI) element displaying sign language video, according to an embodiment of the present disclosure. [Figure 6] A diagram illustrating an example scenario for adjusting the size of a user interface (UI) element displaying sign language video, according to an embodiment of the present disclosure. [Figure 7]1 is a flowchart illustrating example operations for displaying sign language video through adjustable user interface (UI) elements, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008]

[0014] The implementations described below can be found in the disclosed electronic device and method for display of sign language video through adjustable user interface (UI) elements. An exemplary aspect of the present disclosure provides an electronic device (e.g., a smart television or a mobile device) that can be coupled to a display device. The electronic device can receive a media stream that can include a video. The electronic device can determine a position of a signer in the video. The signer can be an animated character or person that can perform using sign language in the video. The electronic device can further extract from the video a video portion that corresponds to the determined position of the signer in the video. The electronic device can control playback of the video on a display device. The electronic device can control the display device based on the playback to render a user interface (UI) element on the display device and display the extracted video portion within the UI element. The UI element can be rendered as a picture-in-picture (PiP) window of adjustable size. The electronic device can thereby provide an adjustable UI element for convenient viewing of the signer's video together with a main video.

[0009]

[0015] In an embodiment, the electronic device can receive metadata associated with the video. The metadata can include information describing a position of a signer in the video at a number of timestamps. The electronic device can determine a position of the signer in the video based on the received metadata. In another embodiment, the electronic device can detect an area in the video based on a difference between a background of the area in the video and a background of a remainder of the video. The electronic device can detect a position of the signer in the video based on the detection of the area. In another embodiment, the electronic device can detect a border around the signer in the video. The electronic device can detect a position of the signer in the video based on the detection of the border. In some embodiments, the electronic device can be configured to detect hand signs associated with a sign language in the video based on application of a neural network model to one or more frames of the video (e.g., a live video broadcast). The electronic device can be configured to detect a position of the signer in the video based on the detection of the hand signs. The electronic device can further extract a video portion of the signer and control a display device to render a UI element (e.g., a PiP window) on the display device based on the detected position of the signer. Thereby, the electronic device can automatically detect the position of a signer for a live video broadcast and generate a PiP window for the signer.

[0010]

[0016] In an embodiment, the electronic device may provide the ability to customize UI elements (e.g., a PiP window) according to user preferences. The electronic device may be configured to adjust a size of a UI element, a position of a UI element, a theme or color scheme of a UI element, a preference to hide a UI element, and a schedule for rendering a UI element based on the user preferences. For example, the electronic device may receive a first input to change a current position of a PiP window to a first position different from the current position. The electronic device may control a display device to render the PiP window at the first position based on the first user input. In another example, the electronic device may receive a second input to change a current size of a PiP window to a first size different from the current size. The electronic device may control a display device to change a current size of the PiP window to match the first size based on the second input. The electronic device may thereby provide a simple and easy-to-use UI technique for adjusting a size and a position of a video portion of a signer based on the PiP window. Based on adjusting the position and size of the PiP window, the electronic device can provide a clear, enlarged view of the signer's video and can allow for an unobstructed view of the primary video (such as important notifications within the primary video).

[0011]

[0017] 1 is a diagram illustrating an example network environment for displaying sign language video through adjustable user interface (UI) elements, according to an embodiment of the present disclosure. With reference to FIG. 1, a diagram of a network environment 100 is shown. Shown in the network environment 100 are an electronic device 102, a display device 104, and a server 106. The electronic device 102 can be communicatively coupled to the server 106 via a communication network 108. The electronic device 102 can be communicatively coupled to the display device 104 directly or via the communication network 108. With reference to FIG. 1, further shown is a main video 104A and a UI element 112 displaying a video of a signer 110.

[0012]

[0018] 1, the electronic device 102 and the display device 104 are shown as two separate devices. However, in some embodiments, the display device 104 can be integrated with the electronic device 102 without departing from the scope of the present disclosure.

[0013]

[0019] The electronic device 102 may include suitable logic, circuitry, and interfaces that may be configured to render user interface (UI) elements 112 on the display device 104. The electronic device 102 may receive a first media stream including a video. The electronic device 102 may further receive metadata associated with the video. The electronic device 102 may further extract a video portion associated with a position of the signer 110 in the video. The electronic device 102 may control the display device 104 to display the extracted video portion in the UI element 112. The electronic device 102 may include suitable middleware and codecs capable of receiving and / or playing media content (e.g., video). In an embodiment, the electronic device 102 may be associated with multiple user profiles. Each user profile of the multiple user profiles may include a collection of content items, settings or menu options, user preferences, and the like. The electronic device 102 may enable selection and / or switching of a user profile among the multiple user profiles on a graphical user interface based on user input from a remote control or a touch screen interface. The electronic device 102 may include an infrared receiver or a Bluetooth interface to receive control signals transmitted from a remote control in response to buttons being pressed on the remote control.Examples of electronic devices 102 may include, but are not limited to, smart televisions (TVs), Internet Protocol TVs (IPTVs), digital media players, microconsoles, set-top boxes, over-the-top (OTT) players, streaming players, media extenders / regulators, digital media hubs, smartphones, personal computers, laptops, tablets, wearable electronic devices, head-mounted devices, or any other display device capable of receiving, decoding, and playing content from broadcast signals over cable or satellite networks, streaming content broadcast over the air, content from Internet-based communication signals, etc. Examples of content may include, but are not limited to, images, animation (such as 2D / 3D animation or motion graphics), audio / video data, traditional television programming (delivered via traditional broadcast, cable, satellite, Internet, or other means), pay-per-view programming, on-demand programming (such as in a video-on-demand (VOD) system), or Internet content (e.g., streaming media, downloadable media, webcasts, etc.).

[0014]

[0020] In an embodiment, the electronic device 102 can be configured to generate a UI element 112 (e.g., a picture-in-picture (PiP) window) based on the received first media stream. The PiP window can be adjustable in size and position based on user input. The PiP window can display a portion of the primary video 104A that includes a signer 110 performing a sign language within the primary video. The functionality of generating the UI element 112 (e.g., a PiP window) can be integrated with the electronic device 102 by the manufacturer of the electronic device 102 or can be downloadable as an add-on application from the server 106 or an application store / marketplace.

[0015]

[0021] The display device 104 may include suitable logic, circuitry, and interfaces that may be configured to render a UI element 112 that displays the extracted video portion of the signer. The display device 104 may be further configured to display the main video 104A being played by the electronic device 102. In one embodiment, the display device 104 may be an external display device that is connected to the electronic device 102. For example, the display device 104 may be connected to the electronic device 102 (such as a digital media player or personal video recorder) by a wired connection (such as a High-Definition Multimedia Interface (HDMI) connection) or a wireless connection (such as Wi-Fi). In another embodiment, the display device 104 may be integrated with the electronic device 102 (such as a smart television). The display device 104 (such as a display screen with integrated audio speakers) may include one or more controllable parameters, such as brightness, contrast, aspect ratio, color saturation, audio volume, and the like. The electronic device 102 can be configured to control parameters of the display device 104 by sending one or more signals over a wired connection (such as an HDMI connection). In one embodiment, the display device 104 can be a touch screen capable of receiving user input via touch input. The display device 104 can be implemented through a number of known technologies, such as, but not limited to, at least one of liquid crystal display (LCD) display, light emitting diode (LED) display, plasma display, or organic LED (OLED) display technology, or other display devices.In at least one embodiment, the display device 104 may be a display unit of a smart TV, a head mounted device (HMD), a smart glasses device, a see-through display, a head-up display (HUD), an in-vehicle infotainment system, a projected display, an electrochromic display, or a transparent display.

[0016]

[0022] The server 106 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store one or more media streams. The server 106 may further be configured to store metadata for determining a location of a signer in one or more videos. In some embodiments, the server 106 may be configured to train a neural network model for detection of hand gestures associated with a sign language in a video. In some embodiments, the server may be configured to store a neural network model and a training data set for training the neural network model. The server 106 may further be configured to store user profiles associated with the electronic device 102, preferences associated with UI elements 112 for each user profile, usage history of UI elements 112 for each user profile, and the like. The server 106 may be implemented as a cloud server and may perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, and the like. Other implementations of the server 106 may include, but are not limited to, a database server, a file server, a web server, a media server, an application server, a mainframe server, or a cloud computing server. In at least one embodiment, the server 106 can be implemented as multiple distributed cloud-based resources using a number of techniques known to those of skill in the art. Those skilled in the art will appreciate that the scope of the present disclosure cannot be limited to the implementation of the server 106 and the electronic device 102 as two separate entities. In certain embodiments, the functionality of the server 106 can be incorporated in whole or at least in part into the electronic device 102 without departing from the scope of the present disclosure.

[0017]

[0023] The communication network 108 may include a communication medium through which the electronic device 102, the display device 104, and the server 106 may communicate with each other. The communication network 108 may be one of a wired connection or a wireless connection. Examples of the communication network 108 may include, but are not limited to, the Internet, a cloud network, a cellular or wireless mobile network (such as Long-Term Evolution and 5G New Radio), a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 108 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communications, wireless access points (AP), device-to-device communications, cellular communications protocols, and Bluetooth (BT) communications protocols.

[0018]

[0024] In operation, the electronic device 102 can receive a first media stream that can include video. The first media stream can be encoded into a standard digital container format for transmission of video. In an embodiment, the electronic device 102 can receive the first media stream from the server 106. In an embodiment, the electronic device 102 can be further configured to receive metadata associated with the video. The metadata can include information describing a location of the signer 110 in the video at multiple timestamps. The signer 110 can be an animated character or person that can perform using sign language in the video. For example, the signer 110 can be a person that can interpret spoken words in the video and convert the spoken words into sign language.

[0019]

[0025] In an embodiment, the electronic device 102 may be further configured to determine a position of the signer 110 in the video based on receiving the metadata. The determined position may include image coordinates of corners of a rectangular area of ​​the video that includes the signer 110. In another embodiment, the electronic device 102 may be configured to apply a neural network model (shown in FIG. 2) to the video to determine the position of the signer 110 in the video. In another embodiment, the electronic device 102 may determine the position of the signer 110 in the video by image analysis based on a difference between the area around the signer 110 and the background of the main video 104A, or based on detection of a border around the signer 110 in the video. Details of the application of the neural network model and the image analysis are shown, for example, in FIGS. 2 and 3. In an embodiment, the electronic device 102 may be further configured to control the display device 104 to render a highlighted border around the signer 110 in the displayed video. The border may be rendered based on the determined position. In one example, if the electronic device 102 identifies multiple candidates for the video portion of the signer 110 from the video, the electronic device 102 can display a border to obtain user confirmation of the video portion of the signer in the video.

[0020]

[0026] The electronic device 102 may be further configured to extract a video portion that corresponds to the determined position of the signer 110 in the video. The video portion may be extracted from a rectangular region of the video. In an embodiment, the electronic device 102 may receive a second media stream that includes the extracted video portion. The second media stream may be different from the first media stream. The electronic device 102 may be further configured to control playback of the video (e.g., the primary video 104A) on the display device 104. The electronic device 102 may be further configured to control the display device 104 to render a UI element 112 on the display device 104 and display the extracted video portion within the UI element 112. The UI element 112 may be rendered as a picture-in-picture (PiP) window of adjustable size and position.

[0021]

[0027] In an embodiment, the electronic device 102 can be configured to customize the UI element 112 (PiP window) according to a user preference. For example, the electronic device 102 can be configured to adjust a size of the UI element 112, a position of the UI element 112, a theme or color scheme of the UI element 112, a preference to hide the UI element 112, and a schedule for rendering the UI element 112 based on the user preference. For example, the electronic device 102 can receive a first input to change a current position of the PiP window to a first position different from the current position. The electronic device 102 can control the display device 104 to render the PiP window at the first position based on the first user input. In another example, the electronic device 102 can receive a second input to change a current size of the PiP window to a first size different from the current size. The electronic device 102 can control the display device to change a current size of the PiP window to match the first size based on the second input. Thereby, the electronic device 102 can provide a simple and easy-to-use UI technique for adjusting the size and position of the signer's video portion based on the UI elements 112 (e.g., a PiP window). Based on adjusting the position and size of the UI elements 112, the electronic device 102 can provide a clear and enlarged view of the signer's video and can enable an unobstructed view of the primary video 104A (e.g., an important notification in the primary video 104A).

[0022]

[0028] Modifications, additions, or omissions may be made to Figure 1 without departing from the scope of this disclosure. For example, network environment 100 may include more or fewer elements than those illustrated and described in this disclosure.

[0023]

[0029] FIG. 2 is a block diagram of an example electronic device for displaying sign language videos through adjustable user interface (UI) elements, according to an embodiment of the disclosure. The description of FIG. 2 is provided with reference to the elements of FIG. 1. Referring to FIG. 2, a block diagram 200 of an electronic device 102 is shown. The electronic device 102 can include a circuit 202, a memory 204, an input / output (I / O) device 206, a network interface 208, and a neural network model 210. In at least one embodiment, the electronic device 102 can also include a display device 104. The circuit 202 can be communicatively coupled to the memory 204, the I / O device 206, the network interface 208, the neural network model 210, and the display device 104.

[0024]

[0030] The circuitry 202 may include suitable logic, circuits, and interfaces that may be configured to execute program instructions associated with different operations to be performed by the electronic device 102. The different operations include determining a position of a signer in a video, extracting a video portion corresponding to the determined position of the signer in the video, and controlling the display device 104 to render a UI element 112 on the display device 104 and display the extracted video portion in the UI element 112. The circuitry 202 may include one or more processing units, which may be implemented as an integrated processor or a group of processors that jointly perform the functions of the one or more processing units. The circuitry 202 may be implemented based on several processor technologies known in the art. Examples of implementations of the circuitry 202 may be an x86-based processor, a graphics processing unit (GPU), a reduced instruction set computer (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computer (CISC) processor, a microcontroller, a central processing unit (CPU), and / or other computer circuitry.

[0025]

[0031] The memory 204 may include suitable logic, circuits, and interfaces that may be configured to store program instructions to be executed by the circuitry 202. In an embodiment, the memory 204 may store the received first media stream, the second media stream, the received metadata, the determined location of the signer 110, and the extracted video portion. The memory 204 may further be configured to store one or more user profiles associated with the electronic device 102, preferences associated with the UI elements 112 for each user profile, a usage history of the UI elements 112 for each user profile, a sign language preference (e.g., American Sign Language or British Sign Language) for each user profile, and the like. In some embodiments, the memory 204 may further store one or more preset locations and one or more preset sizes of the UI elements 112. The memory 204 may store one or more pre-defined positions and one or more pre-defined sizes of the UI elements 112 as a default for all users, or may store one or more pre-defined positions and one or more pre-defined sizes of the UI elements 112 for each user profile. The memory 204 may be further configured to store pre-defined templates for image analysis, the neural network model 210, and training data sets received from the server 106. Examples of implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0026]

[0032] The I / O devices 206 may include suitable logic, circuits, and interfaces that may be configured to receive one or more inputs and provide one or more outputs based on the received one or more inputs. The I / O devices 206, including various input and output devices, may be configured to communicate with the circuitry 202. In one example, the electronic device 102 may receive, via the I / O devices 206, a user input indicating a change in a current position of a UI element 112 rendered on the display device 104. In another example, the electronic device 102 may receive, via the I / O devices 206, a user input indicating a change in a current size of a UI element 112 rendered on the display device 104. Examples of the I / O devices 206 may include, but are not limited to, a remote console, a touch screen, a keyboard, a mouse, a joystick, a microphone, a display device (such as the display device 104), and a speaker.

[0027]

[0033] The network interface 208 may include suitable logic, circuits, and interfaces that may be configured to facilitate communication between the circuitry 202 and the server 106 or the display device 104 over the communications network 108. The network interface 208 may be implemented using various known technologies that support wired or wireless communication between the electronic device 102 and the communications network 108. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, a Bluetooth receiver, an infrared receiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuit. The network interface 208 may be configured to communicate via wireless communication with a network, such as the Internet, an intranet, or a wireless network, such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN). The wireless communication may use one or more of a number of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), email protocols, instant messaging, and short message service (SMS).

[0028]

[0034] The neural network model 210 may be a computational network or system of artificial neurons arranged as nodes in layers. The layers of the neural network model may include an input layer, one or more hidden layers, and an output layer. Each layer of the layers may include one or more nodes (or artificial neurons, e.g., represented by circles). The output of every node in the input layer may be coupled to at least one node in the hidden layer(s). Similarly, the input of each hidden layer may be coupled to the output of at least one node in another layer of the neural network model. The output of each hidden layer may be coupled to the input of at least one node in another layer of the neural network model. The node(s) in the final layer may receive inputs from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer may be determined from hyperparameters of the neural network model. Such hyperparameters may be set before, during, or after training the neural network model 210 on a training data set.

[0029]

[0035] Each node of the neural network model 210 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the network. The set of parameters may include, for example, weight parameters, regularization parameters, etc. Each node may use the mathematical function to calculate an output based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the neural network model 210. All or some of the nodes of the neural network model 210 may correspond to the same or different mathematical functions.

[0030]

[0036] According to an embodiment, the circuitry 202 can train the neural network model 210 with one or more features associated with the video, one or more features associated with a background of the signer 110 in the video, one or more features associated with hand movements of the signer 110 in the video, etc., to obtain a trained neural network model 210. The neural network model 210 can be trained to detect hand gestures associated with a sign language in the video and detect a location of the signer 110 in the video based on the detection of the hand gestures. In another embodiment, the neural network model 210 can be trained to distinguish a background of the signer 110 in the video from other parts of the video and detect a location of the signer 110 in the video based on the background. For example, the circuitry 202 can input a video, predefined hand gestures of a sign language (e.g., American Sign Language or British Sign Language), etc., to train the neural network model 210.

[0031]

[0037] In training the neural network model 210, one or more parameters of each node of the neural network model can be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on a loss function for the neural network model. The above process can be repeated for the same or different inputs until a minimum of the loss function can be reached and the training error can be minimized. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, metaheuristics, etc.

[0032]

[0038] The neural network model 210 may include electronic data that may be implemented, for example, as a software component of an application executable on the electronic device 102. The neural network model 210 may rely on libraries, external scripts, or other logic / instructions executed by a processing device, such as the circuit 202. The neural network model 210 may include code and routines configured to enable a computing device, such as the circuit 202, to perform one or more operations for detection of hand gestures associated with sign language in a video. Additionally or alternatively, the neural network model 210 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the performance of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network model 210 may be implemented using a combination of hardware and software.

[0033]

[0039] Examples of the neural network model 210 may include, but are not limited to, a deep neural network (DNN), a convolutional neural network (CNN), R-CNN, Fast R-CNN, Faster R-CNN, an artificial neural network (ANN), a (You Only Look Once) YOLO network, a CNN+ANN, a fully connected neural network, a deep Bayesian neural network, and / or a combination of such networks. In certain embodiments, the neural network model 210 may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0034]

[0040] The image analysis processor 212 may include suitable hardware and software algorithms that may be configured to perform one or more image analysis techniques, such as object detection, object recognition, image segmentation, motion detection, pose estimation, edge detection, template matching, etc. For example, the image analysis processor 212 may perform template matching based on predefined features or templates associated with the shape and size of portions of the signer 110 in the video. In another example, the image analysis processor 212 may perform edge detection to detect a visible boundary around the signer 110. In another example, the image analysis processor 212 may perform motion detection to distinguish between a non-moving (static) area behind the signer 110 and a moving (dynamic) background of the main video 104A.

[0035]

[0041] The functions or operations performed by the electronic device 102 as described in Figure 1 may be performed by the circuitry 202. The operations performed by the circuitry 202 are described in detail, for example, in Figures 3, 4, 5, 6 and 7.

[0036]

[0042] FIG. 3 illustrates an example scenario for displaying sign language video through adjustable user interface (UI) elements based on metadata, according to an embodiment of the present disclosure. FIG. 3 is described in conjunction with elements of FIG. 1 and FIG. 2. With reference to FIG. 3, an example scenario 300 is illustrated. In the example scenario 300, a block of received data 302 is illustrated. The received data 302 may include a first media stream 304, which may include one or more frames of a video, and metadata 306 associated with the video. The video may include a signer 308. The one or more frames may include a first frame 304A, a second frame 304B, ..., an Nth frame 304N. With reference to FIG. 3, a display device 104 associated with the electronic device 102 is also illustrated. The display device 104 may be configured to display one or more frames of a video (e.g., a main video 310).

[0037]

[0043] The first media stream 304 may include one or more frames of video that may be rendered on the display device 104. For example, a primary video 310 may be encapsulated in the first media stream 304. In one embodiment, the circuit 202 may receive the first media stream 304 from a server 106. In another embodiment, the circuit 202 may receive the first media stream 304 from a server associated with a broadcast network. In such a scenario, the first media stream 304 may include text information, such as an electronic program guide (EPG), associated with a broadcast channel.

[0038]

[0044] The metadata 306 may include information associated with the video and may be embedded in an analog or digital signal of the video. As an example, the metadata 306 may include information associated with a duration of the video, a title of the video, a resolution of the video, a codec and / or container type associated with the video, information of one or more characters 312 or persons in the video (e.g., the main video 310), and the like. In an embodiment, the metadata 306 may include information associated with a signer 308 present in the video. The signer 308 may be an animated character or person that may perform using sign language in the video. In an embodiment, the signer 308 may translate a language (e.g., English) spoken by one or more characters 312 in the video into a sign language (e.g., American Sign Language (ASL)). The metadata 306 may include information that may describe a location of the signer 308 in the video at multiple timestamps. The location of the signer 308 may include image coordinates 314 that may correspond to corners of a rectangular area of ​​the video that includes the signer 308. In one embodiment, the image coordinates 314 may be labeled with respect to pixels or image coordinates of one or more frames of a video. Table 1 below shows examples of metadata 306 received at different timestamps. Metadata indicating the position of the signer in the video at different timestamps [Table 1]

[0039]

[0045] In an embodiment, the circuit 202 can be configured to determine a position of the signer 308 at a number of timestamps based on the received metadata 306. For example, the circuit 202 can be configured to analyze the received metadata 306 to determine a position of the signer 308. With reference to Table 1, the position of the signer 308 in the video at a number of timestamps listed in the first column of Table 1 can be determined from the third column of Table 1. Table 1 (third column) includes a set of four image coordinates that indicate a rectangular boundary of the signer 308 in the video, although the disclosure is not limited in this respect. The metadata 306 can include any number of coordinates depending on the shape (e.g., polygonal shape) of the boundary of the signer 308.

[0040]

[0046] The circuit 202 can be further configured to extract a video portion 316 from the video. The extracted video portion 316 can correspond to the determined location in the video. For example, the video portion 316 can be extracted from a rectangular region of the video that corresponds to an area between the image coordinates 314 and includes the signer 308. The circuit 202 can be further configured to control playback of the video on the display device 104. In an embodiment, the circuit 202 can be configured to control playback of the video on the display device 104 based on user input. In an embodiment, the circuit 202 can be configured to control the display device 104 to render a border around the signer 308 in the video based on the extracted video portion 316. The display device 104 can render the border based on the determined location of the signer 308 in the video. A border around the signer 308 can be rendered to distinguish the signer 308 from one or more characters 312 in the video. In some embodiments, the circuitry 202 may highlight a border around the signer 308 in a bright color (e.g., bright green) to assist the user in locating the signer 308 within the video.

[0041]

[0047] The circuitry 202 can be configured to control the display device 104 based on the playback to render user interface (UI) elements 318 on the display device 104. The UI elements 318 can be rendered as picture-in-picture (PiP) windows. The PiP windows can have an adjustable size. For example, the circuitry 202 can be configured to adjust the size of the PiP window based on one or more user inputs. The circuitry 202 can further be configured to display the extracted video portions 316 in the UI elements 318 (e.g., the PiP window). The extracted video portions 316 can include the signer 308. In an embodiment, the circuitry can connect the image coordinates 314 with lines and copy the video portions 316 within those lines into the PiP window in real time.

[0042]

[0048] In an embodiment, the circuit 202 can be configured to receive a second media stream (e.g., a second signal) from the server 106, which can be different from the first media stream 304. For example, the second media stream can include a video portion 316. In such a scenario, the circuit 202 can be configured to control playback of the video from the first media stream 304 on the display device 104. The circuit 202 can be further configured to control the display device 104 to render a UI element 318 on the display device 104. The circuit 202 can control the display device 104 to display the video portion 316 extracted from the received second stream in the UI element 318 in time synchronization with the playback of the video from the first media stream 304. The circuit 202 can be configured to adjust the size of the PiP window in both cases, i.e., when the video portion 316 is extracted based on the received metadata 306 and when the video portion is received as the second media stream from the server 106.

[0043]

[0049] In an embodiment, the circuit 202 may be further configured to receive a first input to change a current position of the rendered UI element 318 (e.g., a PiP window) to a first position 320, which may be different from the current position. The circuit 202 may also receive a second input to change a current size of the rendered UI element 318 (e.g., a PiP window) to a first size, which may be different from the current size. The circuit 202 may be further configured to control the display device 104 to render the UI element 318 at the first position 320 and at the first size based on the first input and the second input. Details of adjusting the position and size of the UI element 318 are shown, for example, in FIGS. 5 and 6. In an embodiment, the circuit 202 may be configured to control the display device 104 to blur the signer 308 in the primary video 310 when a UI element 318 (e.g., a PiP window) including the signer 308 is displayed. In another embodiment, the circuitry 202 can be configured to control the display device 104 to replace a video portion of the signer 308 in the primary video 310 with background pixels of the primary video when a UI element 318 including the signer 308 is displayed.

[0044]

[0050] FIG. 4 illustrates an exemplary scenario for displaying sign language video through adjustable user interface (UI) elements for live video broadcasting, according to an embodiment of the present disclosure. FIG. 4 is described with reference to elements of FIGS. 1-3. Referring to FIG. 4, an exemplary scenario 400 is illustrated. In the exemplary scenario 400, a block of received data 402 is illustrated. The received data 402 may include a third media stream 404 including one or more frames of the live video broadcast. Further illustrated in the exemplary scenario 400 is a signer 406 that may be present in the live video broadcast. The one or more frames may include a first frame 404A, a second frame 404B, ..., an Nth frame 404N. Further illustrated in the exemplary scenario 400 are a neural network model 210 and a display device 104 associated with the electronic device 102.

[0045]

[0051] The third media stream 404 may include a live video broadcast, which may be broadcast in various media, such as terrestrial or over-the-air broadcast, streaming broadcast, satellite television broadcast, etc. For example, the live video broadcast may be encapsulated in the third media stream 404. The live video broadcast may include a main video 408 representing one or more characters 410. In an embodiment, the third media stream 404 may be received from the server 106, or a server associated with a broadcast network. If the third media stream 404 is a live video broadcast, the metadata embedded with the third media stream 404 may not include the location of the signer 406 within the live video broadcast. In such a case, the circuit 202 may be configured to analyze frames of the live video broadcast to determine the location of the signer 406 within the live video broadcast. For example, the circuit 202 may be configured to perform image analysis on one or more frames of the live video broadcast to detect the area of ​​the signer 406 within the video (e.g., a non-changing background area). In another example, the circuit 202 can be configured to apply the neural network model 210 to one or more frames of the live video broadcast to detect hand gestures associated with sign language in the video. For example, the circuit can be configured to apply the neural network model 210 to a first frame 404A to detect hand gestures in the first frame 404A. The circuit 202 can be configured to apply the neural network model 210 to a second frame 404B, ..., Nth frame 404N to detect hand gestures in each frame of the video. In another embodiment, the circuit 202 can be configured to apply the neural network model 210 to one or more frames of the live video broadcast to distinguish between a background in a portion of the video in which a signer 406 may be present and other portions of the video.The neural network model 210 can be configured to predict a bounding box for the position of the signer 406 in the video at multiple timestamps based on detection of hand gestures or background corresponding to the position of the signer 406 in the video.

[0046]

[0052] The neural network model 210 may be a pre-trained model that may be trained to detect hand gestures associated with a sign language (such as American Sign Language (ASL) or British Sign Language (BSL)) and output image coordinates 412 that correspond to corners of a rectangular region of a video portion that includes the signer 406. The circuitry 202 may be further configured to detect a position of the signer 406 in the video further based on the image coordinates 412. Thereby, the electronic device 102 may identify a position of the signer 406 in the video even if metadata embedded in the third media stream 404 does not include a position of the signer 406 or even if no metadata is present in the third media stream 404.

[0047]

[0053] In another embodiment, the circuit 202 can be configured to perform image analysis (such as object detection) using the image analysis processor 212 to detect the signer 406 in the video and output image coordinates 412. The circuit 202 can detect the area around the signer 406 in the video based on a difference in background color of the area compared to the primary video 408, a difference in background shading of the area compared to the primary video 408, or a predefined boundary around the signer. For example, the circuit 202 can be configured to detect the location of the signer 406 in the video (such as the primary video 408) when a portion of the video that includes the signer has a different background color (or shading) than the background color (or shading) of other portions of the video. In another example, the circuit 202 can be configured to detect a non-moving (static) background area that is different from the background of the primary video 408 to detect the location of the signer 406. In another example, the circuitry 202 can use edge detection or template matching techniques to detect a border around the signer 406 if the video includes a visible boundary of a predetermined shape and color around the signer 406, and can detect the position of the signer 406 within the video based on the detected border. In these scenarios, the circuitry 202 can rely on image analysis techniques that require less computational power compared to running the neural network model 210.

[0048]

[0054] In one embodiment, the circuitry 202 may be further configured to control the display device 104 to render a highlighted border 406A around the signer 406 in the displayed video. The border 406A may be rendered based on the image coordinates 412 of the predicted bounding box. In one example, the circuitry 202 may display the border 406A to obtain user confirmation of the video portion of the signer 406 in the video if the neural network model 210 identifies multiple candidates for the video portion of the signer 406 in the video. For example, the circuitry 202 may display a prompt (press OK to confirm, press right arrow to view next candidate) displayed on the display device 104. TIFF2024542064000003.tif6150.) may receive user confirmation for the highlighted candidate. The circuit 202 may request user confirmation of the signer 406 in the video if the confidence score of the detection and / or bounding box prediction of the signer 406 may be lower than a threshold score. In another embodiment, the circuit 202 may request user confirmation of the signer 406 in the video if the image analysis processor 212 outputs multiple candidates for the video portion of the signer 406 in the video.

[0049]

[0055] The circuit 202 can be further configured to extract a video portion 414 from the live video broadcast. In an embodiment, the circuit 202 can be further configured to extract a video portion 414 from the live video broadcast based on a user confirmation of the highlighted candidate. The extracted video portion 414 can correspond to a determined location within the live video broadcast. For example, the video portion 414 can be extracted from a rectangular region of the video. The rectangular region can correspond to an area between image coordinates 412 and can include the signer 406.

[0050]

[0056] The circuit 202 can be further configured to control playback of the video on the display device 104. In an embodiment, the circuit 202 can be configured to control playback of the video on the display device 104 based on user input. The circuit 202 can be configured to control the display device 104 based on the playback. The circuit 202 can control the display device 104 to render a user interface (UI) element 416 on the display device 104. For example, the UI element 416 can be rendered as a picture-in-picture (PiP) window of adjustable size. The circuit 202 can be configured to display an extracted video portion 414 including the signer 406 in the UI element 416.

[0051]

[0057] FIG. 5 illustrates an exemplary scenario for adjusting a position of a user interface (UI) element displaying a sign language video, according to an embodiment of the present disclosure. FIG. 5 is described with reference to elements of FIGS. 1-4. With reference to FIG. 5, an exemplary scenario 500 is illustrated. In the exemplary scenario 500, an electronic device 102 and a display device 104 associated with the electronic device 102 are illustrated. The electronic device 102 can control the display device 104 to display a primary video 502 within a display area 506. With reference to FIG. 5, a user 508 associated with the electronic device 102 is further illustrated.

[0052]

[0058] In an embodiment, the circuit 202 can be configured to receive user input including a selection of a user profile associated with the user 508. Based on the selected user profile, the circuit 202 can retrieve one or more user preferences to be associated with a user interface (UI) element 510 that can display an extracted video portion 512 of the signer 516. In some embodiments, the circuit 202 can retrieve the one or more user preferences from the memory 204. The UI element 510 can be rendered based on the retrieved one or more user preferences. For example, the one or more user preferences can include one or more of a preference for a position of the UI element 510 within the display area 506 of the display device 104, a theme or color scheme of the UI element 510, a preference for a size of the UI element 510, a preference for showing / hiding the UI element 510, a schedule for rendering the UI element 510, and a sign language preference (e.g., American Sign Language or British Sign Language).

[0053]

[0059] The position preference may include a preferred position at which the UI element 510 may be displayed. The circuit 202 may retrieve a user preference for a first position 514 from the set of positions. The first position 514 may be a preferred position for display of the UI element 510 according to a user profile of the user 508. As an example, the first position 514 may correspond to a bottom right corner within the display area 506 of the display device 104. The theme or color scheme of the UI element 510 may correspond to a design or color preference of a selected user profile of the UI element 510. As an example, the theme or color scheme of the UI element 510 may include a green background behind the signer 516 or a green border of the UI element 510. The size preference of the UI element 510 may include a default size predetermined by a manufacturer of the electronic device 102. The preference to hide the UI element 510 may correspond to a preference of the user 508 as to whether the UI element 510 should be hidden or displayed. The schedule for rendering the UI element 510 can correspond to a first period during which the UI element 510 can be rendered and a second period during which the UI element 510 cannot be rendered. For example, a user preference for the schedule can indicate that the UI element 510 can be rendered between 10:00 a.m. and 4:00 p.m. and can be hidden between 4:01 p.m. and 10:00 p.m. In another embodiment, the user preference can include instructions to display the UI element 510 if one of the characters 504 in the primary video 502 is speaking and to hide the UI element 510 if there is no speech in the primary video 502.

[0054]

[0060] At time T1, the circuit 202 may receive a first media stream that may include a video. The video may include a main video 502 representing a character 504. The circuit 202 may further receive metadata associated with the video. The circuit 202 may further determine a position of a signer 516 in the video based on the received metadata. The metadata may include information describing the position of the signer 516 in the video at multiple timestamps. In another embodiment, the circuit 202 may determine the position of the signer 516 based on image analysis by the image analysis processor 212 or based on application of a neural network model 210. The circuit 202 may further extract a video portion 512 associated with the determined position of the signer 516 in the video. Based on the extracted position, the circuit 202 may control the playback of the video on the display device 104. The circuit 202 may further control the display device 104 to render a UI element 510 (e.g., a PiP window) at a first position 514 on the display device 104 based on the retrieved user preference. The circuit 202 may control the display device 104 to display the extracted video portion 512 in the UI element 510. In some embodiments, if a user preference for a position of the UI element 510 is not available in the memory 204, the UI element 510 may overlap a determined position of the signer 516 in the main video 502 according to a default position predetermined by a manufacturer of the electronic device 102. As shown in FIG. 5, the circuit 202 may control the display device 104 to render the UI element 510 at a lower right corner of the display area 506 of the display device 104.

[0055]

[0061] The circuit 202 can receive a first input 518 to change the current position (or first position 514) of the rendered UI element 510 to a second position 520. The second position 520 can be different from the first position 514. If the electronic device 102 is a television controlled by a remote control, the display device 104 can display a pop-up menu 510A (e.g., a context menu) upon selection of the UI element 510. The pop-up menu 510A can include a "Resize" and a "Move" option. Upon selection of the "Move" option (selection highlighted in gray), the display device 104 can display sub-options "Move to preset position" and "Drag". Upon selection of the "Move to preset position" option (selection highlighted in gray), the display device 104 can display sub-options such as "Preset position 1", "Preset position 2", etc. based on stored preferences of the selected user profile and / or default positions set by the manufacturer of the electronic device 102. For example, "preset position 1" may correspond to the bottom left corner of the display area 506, and "preset position 2" may correspond to the top left corner of the display area 506. These preset positions may be stored in the memory 204 based on set preferences of a selected user profile and / or default positions set by the manufacturer of the electronic device 102. Upon selection of one of the sub-options, the circuit 202 may control the display device 104 to display the UI element 510 in the second position 520 at time T2. As an example, the second position 520 may correspond to the bottom left corner within the display area 506 of the display device 104. Upon selection of the "drag" option, the display device 104 may highlight the UI element to indicate that the UI element 510 has been selected, and may display the UI element 510 in the second position 520 at time T2. TIFF2024542064000004.tif6150) may be used to display a prompt to drag the UI element 510 to any arbitrary position within the display area 506. If the electronic device 102 is a smartphone including a touch screen input, upon selection of the UI element 510, the display device 104 may display a prompt to drag and move the UI element 510 to any arbitrary position within the display area 506 via touch input. The circuitry 202 may control the display device 104 to display the UI element 510 in a second position 520 (e.g., the bottom left corner) at time T2 based on the first input 518. The display device 104 may seamlessly continue playing the extracted video portion 512 of the signer 516 prior to, during, and following the movement of the UI element 510 in time synchronization with the main video 502.

[0056]

[0062] 6 illustrates an example scenario for adjusting the size of a user interface (UI) element displaying a sign language video, according to an embodiment of the present disclosure. An example scenario 600 is illustrated in FIG. 6. In the example scenario 600, an electronic device 102 and a display device 104 associated with the electronic device 102 are illustrated. The electronic device 102 can control the display device 104 to display a primary video 602 within a display area 606.

[0057]

[0063] In an embodiment, the circuit 202 can be configured to receive user input including a selection of a user profile. Based on the selected user profile, the circuit 202 can retrieve one or more user preferences to be associated with a UI element 610 that can display an extracted video portion of the signer 608. In some embodiments, the circuit 202 can retrieve the one or more user preferences from the memory 204. The UI element 610 can be rendered based on the retrieved one or more user preferences. For example, the one or more user preferences can include a preference for a size of the UI element 610.

[0058]

[0064] At time T1, the circuit 202 may receive a first media stream that may include a video. The video may include a main video 602 representing a character 604. The circuit 202 may further receive metadata associated with the video. The circuit 202 may further determine a location of a signer 608 in the video based on the received metadata. In another embodiment, the circuit 202 may determine a location of the signer 608 based on image analysis by an image analysis processor 212 or based on application of a neural network model 210. The circuit 202 may further extract a video portion that may correspond to the determined location in the video. Based on the extracted location, the circuit 202 may be configured to control playback of the video on the display device 104. The circuit 202 may be further configured to control the display device 104 to render a user interface (UI) element 610 (e.g., a PiP window, etc.) on the display device 104 at a first size (e.g., height H1, width W1) based on the retrieved user preferences associated with the selected user profile. In some embodiments, if a user preference for a size of the UI element 610 is not available in the memory 204, the UI element 610 may be displayed based on a default size predetermined by a manufacturer of the electronic device 102. As shown in FIG. 6, the circuitry 202 may control the display device 104 to render the UI element 610 at a first size (H1, W1) within a display area 606 of the display device 104.

[0059]

[0065] The circuit 202 can receive a second input 612 to change the current size (or first size) of the rendered UI element 610 to a second size. The second size can be different from the first size. If the electronic device 102 is a television controlled by a remote control, the display device 104 can display a pop-up menu 610A upon selecting the UI element 610. The pop-up menu 610A can include a "Resize" and a "Move" option. Upon selecting the "Resize" option (selection highlighted in gray), the display device 104 can display a "Resize to Preset Size" and a "Zoom in / Zoom out" sub-options. Upon selecting the "Resize to Preset Size" option (selection highlighted in gray), the display device 104 can display sub-options such as "Preset Size 1", "Preset Size 2", etc. based on retrieved preferences of the selected user profile and / or default sizes set by the manufacturer of the electronic device 102. For example, "preset size 1" and "preset size 2" may correspond to different sizes having a constant aspect ratio so that the extracted video portion of signer 608 has optimal resolution. Upon selection of one of the sub-options, circuit 202 may control display device 104 to display the UI element at a second size (height H2, width W2) at time T2. Upon selection of the "Zoom In / Zoom Out" option, display device 104 may highlight the UI element to indicate that UI element 610 has been selected, and may select the size of the UI element by pressing the arrow buttons ( TIFF2024542064000005.tif6150) to display a prompt to resize the UI element 610 to any arbitrary size within the display area 606. If the electronic device 102 is a smartphone including a touch screen input, upon selecting the UI element 610, the display device 104 can display a prompt to resize the UI element 610 to any arbitrary size within the display area 606 using a touch-based action (such as a pinch open or pinch close action of fingers). The circuit 202 can control the display device 104 to display the UI element at a second size (H2, W2) at time T2 based on the second input 612. For example, the circuit 202 can control the display device 104 to change the current size of the UI element 610 to match the second size (H2, W2). In an embodiment, the circuit 202 can be configured to upscale or downscale the video portion to match the second size (H2, W2) of the UI element 610 before the video portion is displayed within the UI element 610. The circuit 202 can upscale or downscale to change the resolution of the extracted video portion of the signer 608 according to the modified size of the UI element 610. As shown in FIG. 6, the second size (H2, W2) of the UI element 610 can be larger than the first size (H1, W1) of the UI element 610. In such a case, the circuit 202 can upscale the extracted video portion of the signer 608 from a lower resolution (e.g., 720p) to a higher resolution (e.g., 1080p).

[0060]

[0066] FIG. 7 is a flowchart illustrating example operations for displaying sign language video through adjustable user interface (UI) elements, according to an embodiment of the present disclosure. The description of FIG. 7 is provided with reference to elements of FIGS. 1-6. With reference to FIG. 7, a flowchart 700 is shown. Operations 702-712 may be implemented by any computer system, such as by electronic device 102 of FIG. 1 or circuitry 202 of FIG. 2. Operations may begin at 702 and proceed to 704.

[0061]

[0067] At 704, a first media stream including the video can be received. In at least one embodiment, the circuit 202 can be configured to receive the first media stream including the video, for example, as described in Figures 1, 3, and 4.

[0062]

[0068] At 706, a position of the signer 110 within the video can be determined, where the signer 110 can be an animated character or person performing a sign language within the video. In at least one embodiment, the circuitry 202 can be configured to determine a position of the signer 110 within the video. Details regarding determining the position of the signer 110 are shown in Figures 1, 3, and 4.

[0063]

[0069] At 708, a video portion may be extracted from the video that corresponds to the determined location within the video. In at least one embodiment, the circuit 202 may be configured to extract, from the video, a video portion that corresponds to the determined location within the video. Details regarding the extraction of the video portion are shown, for example, in Figures 1, 3, and 4.

[0064]

[0070] At 710, playback of the video on the display device 104 can be controlled. In at least one embodiment, the circuit 202 can be configured to control playback of the video on the display device 104.

[0065]

[0071] At 712, the display device 104 may be controlled based on the playback to render user interface (UI) elements 112 on the display device 104 and display the extracted video portions within the UI elements 112. In at least one embodiment, the circuitry 202 may be configured to control the display device 104 based on the playback to render the UI elements 112 on the display device 104 and display the extracted video portions within the UI elements 112. More details on rendering the UI elements 112 are shown, for example, in Figures 1, 3, 4, and 5. Control may proceed to an end.

[0066]

[0072] Various embodiments of the disclosure may provide a non-transitory computer-readable medium and / or storage medium having instructions stored thereon that a machine and / or computer may execute to operate an electronic device (such as electronic device 102). The instructions may cause the machine and / or computer to perform operations including receiving a first media stream including a video. The operations may further include determining a location of a signer (such as signer 110) in the video. The signer may be an animated character or person that may perform using sign language in the video. The operations may further include extracting from the video a video portion that may correspond to the determined location in the video. The operations may further include controlling playback of the video on a display device (such as display device 104). The operations may further include controlling display device 104 based on the playback to render user interface (UI) elements (such as UI elements 112) on display device 104.

[0067]

[0073] An example aspect of the disclosure can include an electronic device (such as electronic device 102 of FIG. 1 ) that includes circuitry (such as circuit 202) that can be communicatively coupled to a display device (such as display device 104). In an embodiment, electronic device 102 can be configured to receive a first media stream that includes a video. Signer 110 can be an animated character or person performing sign language within the video. Electronic device 102 can be configured to determine a position of signer 110 within the video. The determined position can include image coordinates that correspond to corners of a rectangular region of the video that includes signer 110.

[0068]

[0074] According to an embodiment, the electronic device 102 can receive metadata associated with the video. The metadata includes information describing the location of the signer 110 within the video at multiple timestamps. The electronic device 102 can determine the location of the signer 110 within the video based on the received metadata.

[0069]

[0075] According to an embodiment, the electronic device 102 may be configured to detect hand gestures associated with a sign language in the video based on application of a neural network model (such as the neural network model 210) to frames of the video. The electronic device 102 may be further configured to detect a location of the signer 110 in the video based on the detection of the hand gestures. In such an embodiment, the video may correspond to a live video broadcast.

[0070]

[0076] According to one embodiment, the electronic device 102 can detect a region in the video based on a difference between a background of the region in the video and a background of the remainder of the video. The electronic device 102 can detect a location of the signer 110 in the video based on the detection of the region. In another embodiment, the electronic device 102 can detect a border around the signer 110 in the video. The electronic device 102 can detect a location of the signer in the video based on the detection of the border.

[0071]

[0077] According to an embodiment, the electronic device 102 can be configured to extract from the video a video portion that corresponds to the determined location in the video. The video portion is extracted from a rectangular region of the video. The electronic device can be further configured to control playback of the video on the display device 104. The electronic device 102 can be further configured to control the display device based on the playback to render a user interface (UI) element (e.g., UI element 112) on the display device 104 and display the extracted video portion within the UI element 112. The UI element 112 can be rendered as a picture-in-picture (PiP) window of adjustable size. In an embodiment, the electronic device 102 can be further configured to control the display device 104 to render a border around the signer 110 in the displayed video based on the determined location.

[0072]

[0078] According to an embodiment, the electronic device 102 can receive a first user input including one or more user preferences to be associated with a UI element. The UI element can be rendered based on the received first user input. The one or more user preferences can include a preference for a position of the UI element 112 within a display area (e.g., display area 506) of the display device 104, a theme or color scheme of the UI element 112, a preference for a size of the UI element 112, a preference for hiding the UI element 112, and a schedule for rendering the UI element 112.

[0073]

[0079] According to an embodiment, the electronic device 102 may be configured to receive a first input (e.g., the first input 518) to change a current position of the rendered UI element 112 to a first position different from the current position. The electronic device 102 may be further configured to control the display device 104 based on the first input to render the UI element 112 at the first position. The first position may be within the display area 506 of the display device 104.

[0074]

[0080] According to an embodiment, the electronic device 102 may be configured to receive a second input (e.g., the second input 612) for changing a current size of the rendered UI element 112 to a first size different from the current size. The electronic device 102 may be configured to control the display device 104 based on the received second input to change the current size of the rendered UI element 112 to match the first size. The electronic device 102 may be further configured to upscale or downscale the video portion to match the first size of the UI element 112 before the video portion is displayed within the UI element 112.

[0075]

[0081] According to an embodiment, the electronic device 102 can be configured to receive a second media stream that includes the extracted video portion. The second media stream can be different from the first media stream.

[0076]

[0082] The present disclosure can be realized in the form of hardware or in the form of a combination of hardware and software. The present disclosure can be realized in a centralized manner in at least one computer system or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other device adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system that includes a computer program that, when loaded and executed, can control a computer system to perform the methods described herein. The present disclosure can be realized in the form of hardware, including parts of an integrated circuit that also perform other functions.

[0077]

[0083] The present disclosure may also be embodied in a computer program product, which includes all features enabling the implementation of the methods described herein and which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system capable of processing information to perform a particular function, either directly or after a) conversion into another language, code or notation, b) reproduction in a different content form, or both.

[0078]

[0084] Although the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the present disclosure. Many modifications can also be made to adapt a particular situation or content to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is not intended that the present disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims. [Explanation of symbols]

[0079] 100 Network Environment 102 Electronic Devices 104 Display Devices 104A Main Video 106 Server 108 Communication Network 110 Sign Language 112 User Interface (UI) Elements 200 Block Diagram 202 Circuit 204 Memory 206 Input / Output (I / O) Devices 208 Network Interface 210 Neural Network Model 212 Image Analysis Processor 300 Example Scenarios 302 Data Received 304 Primary Media Stream 304A First Frame 304B Second Frame 304N Nth frame 306 Metadata 308 Sign Language 310 Main Video 312 1 or more characters 314 Image Coordinates 316 Video Part 318 User Interface (UI) Elements 320 1st position 400 Example Scenarios 402 Data Received 404 Tertiary Media Stream 404A First Frame 404B Second Frame 404N Nth frame 406 Sign Language Users 406A Highlighted Border 408 Main Video 410 1 or 2 characters 412 Image Coordinates 414 Video Part 416 User Interface (UI) Elements 500 Example Scenarios 502 Main Video 504 characters 506 Display Area 508 users 510 User Interface (UI) Elements 510A Pop-up Menu 512 Video Part 514 1st position 516 Sign Language Users 518 First Input 520 2nd position 600 Example Scenarios 602 Main Video 604 characters 606 Display Area 608 Sign Language 610 User Interface (UI) Elements 610A Pop-up Menu 612 Second Input 700 Flowchart 702 Start 704 receives a first media stream that includes a video. 706 Determine the location of the signer in the video, where the signer is an animated character or person performing a sign language in the video. 708 video, extracting a video portion corresponding to a determined position in the video. 710 Controlling video playback on display devices 712 based on the playback, control a display device to render a user interface (UI) element on the display device and display the extracted video portion within the UI element. H1 Height H2 Height T1 Time T2 time W1 width W2 Width

Claims

1. 1. An electronic device comprising: a display device including a display unit, The circuit comprises: Receiving a first media stream including a video; determining a position of a signer within the video, the signer is an animated character or person performing sign language within the video; and extracting from the video a video portion that corresponds to the determined position of the signer within the video; controlling playback of the video on the display device; Controlling the display device based on the playback; Rendering user interface (UI) elements on the display device; Displaying the extracted video portion within the UI element. To make it so, configured to:

1. An electronic device comprising:

2. 10. The electronic device of claim 1, wherein the UI elements are rendered as picture-in-picture (PiP) windows of adjustable size.

3. The circuit comprises: receiving metadata associated with the video, the metadata including information describing the position of the signer within the video at a number of timestamps; determining the position of the signer within the video based on the received metadata; and further configured to:

2. The electronic device according to claim 1 .

4. the determined locations include image coordinates corresponding to corners of a rectangular region of the video that includes the signer; the video portion is extracted from the rectangular region of the video; 2. The electronic device according to claim 1 .

5. The circuit comprises: detecting hand signs associated with the sign language in the video based on application of a neural network model to frames of the video; detecting the position of the signer within the video based on the detection of the hand gestures; Further configured as follows:

2. The electronic device according to claim 1 .

6. 6. The electronic device of claim 5, wherein the video corresponds to a live video broadcast.

7. The circuit comprises: Detecting a region in the video based on a difference between a background of the region in the video and a background of a remainder of the video; detecting the location of the signer within the video based on the detection of the region; Further configured as follows:

2. The electronic device according to claim 1 .

8. The circuit comprises: Detecting a boundary around the signer in the video; detecting the position of the signer within the video based on the detection of the boundary; Further configured as follows:

2. The electronic device according to claim 1 .

9. 10. The electronic device of claim 1, wherein the circuitry is further configured to control the display device to render a border around the signer in the displayed video based on the determined position.

10. The circuit comprises: receiving a first input for changing a current position of the rendered UI element to a first position different from the current position; controlling the display device based on the first input to render the UI element at the first position within a display area of ​​the display device; Further configured as follows:

2. The electronic device according to claim 1 .

11. The circuit comprises: receiving a second input for resizing a current size of the rendered UI element to a first size different from the current size; Controlling the display device based on the received second input to change the current size of the rendered UI element to match the first size; controlling the display device to upscale or downscale the video portion to match the first size of the UI element before the video portion is displayed within the UI element; Further configured as follows:

2. The electronic device according to claim 1 .

12. 10. The electronic device of claim 1, wherein the circuitry is further configured to receive a second media stream that includes the extracted video portion, the second media stream being different from the first media stream.

13. the circuitry is further configured to receive a first user input comprising one or more user preferences associated with the UI element; the UI element is rendered based on the received first user input.

2. The electronic device according to claim 1 .

14. 14. The electronic device of claim 13, wherein the one or more user preferences include one or more of: a preference for a position of the UI element within a display area of ​​the display device; a theme or color scheme of the UI element; a preference for a size of the UI element; a preference to hide the UI element; and a schedule for rendering the UI element.

15. 1. A method comprising: receiving a first media stream comprising a video; determining a position of a signer within the video, the signer is an animated character or person performing sign language within the video; and extracting from the video a video portion that corresponds to the determined position of the signer within the video; controlling playback of said video on a display device; Controlling the display device based on the playback; Rendering user interface (UI) elements on the display device; Displaying the extracted video portion within the UI element. Steps to make it so: The method according to claim 1, further comprising:

16. 16. The method of claim 15, wherein the UI elements are rendered as picture-in-picture (PiP) windows of adjustable size.

17. receiving metadata associated with the video, the metadata including information describing the position of the signer within the video at a number of timestamps; determining the position of the signer within the video based on the received metadata; 16. The method of claim 15 further comprising:

18. the determined locations include image coordinates corresponding to corners of a rectangular region of the video that includes the signer; the video portion is extracted from the rectangular region of the video; 16. The method according to claim 15.

19. detecting hand gestures associated with the sign language in the video based on application of a neural network model to frames of the video; detecting the position of the signer within the video based on the detection of the hand gestures; 16. The method of claim 15 further comprising:

20. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by an electronic device, cause the electronic device to perform an operation, the operation including: Receiving a first media stream including a video; determining a position of a signer within the video, the signer is an animated character or person performing sign language within the video; and extracting from the video a video portion that corresponds to the determined position of the signer within the video; Controlling playback of the video on a display device; Controlling the display device based on the playback; Rendering user interface (UI) elements on the display device; Displaying the extracted video portion within the UI element. To make it so, Including, A non-transitory computer-readable medium comprising:

Citation Information

Patent Citations

  • Program output device, server, output method for program and auxiliary information, and program

    JP2016096451A

  • Program output device, output method and program of program and supplementary information

    JP2016152574A

  • Synthesizer, synthetic method, and program

    JP2019075754A

  • Display device and controlling method thereof

    KR1020170010484A

  • Selective sign language location

    US20160100121A1