Haptic content presentation and implementation

Abstract

The present disclosure relates to haptic content presentation and implementation. A method includes displaying a video comprising video frames on a touchscreen and determining a region of interest in a video frame based on a saliency map of the video frame. The method also includes detecting a touch on a touchscreen region while displaying the video frame and generating a haptic response in response to determining that the touchscreen region overlaps the region of interest.

Description

Technical Field

[0001] This disclosure relates to the presentation and implementation of tactile content. Background Technology

[0002] Hundreds of millions of people watch and enjoy video content every day (e.g., on computers, smartphones, televisions, etc.). However, blind or visually impaired people face challenges when watching videos. Because they cannot see the video completely, other technologies are used to convey what is happening in the video. For example, descriptive audio tracks can be played at different parts of the video to describe the events shown in the video. However, these technologies can interfere with the presentation of the content. For instance, descriptive audio may conflict with the video's regular audio track because it plays on top of the main audio track. Summary of the Invention

[0003] According to an embodiment, a method includes displaying a video comprising video frames on a touchscreen, and determining a region of interest (ROI) within the video frames based on a saliency map of the video frames. The method also includes detecting a touch on a touchscreen area while the video frames are displayed, and generating a haptic response in response to determining that the touchscreen area overlaps with the ROI. Other embodiments include apparatus and systems for performing the method. Attached Figure Description

[0004] The above aspects can be understood and appreciated in detail by referring to the accompanying drawings, which provide a more specific description of the embodiments briefly summarized herein.

[0005] However, it should be noted that the accompanying drawings illustrate typical embodiments and should therefore not be considered limiting; other equally effective embodiments are also considered.

[0006] Figure 1 An example system is shown;

[0007] Figure 2 yes Figure 1 A flowchart of an example method in the system;

[0008] Figure 3 yes Figure 1 A flowchart of an example method in the system;

[0009] Figure 4 It shows Figure 1 Example video frames from the system;

[0010] Figure 5 It shows Figure 1 Example saliency graphs in the system;

[0011] Figure 6 It shows Figure 1 Example touch detection in the system; and

[0012] Figure 7 It shows Figure 1 Example haptic response in the system. Detailed Implementation

[0013] This disclosure describes a system to assist blind and visually impaired individuals in experiencing and appreciating video content. Typically, the system uses saliency maps to identify regions of interest within a video frame, which may correspond to objects within the video frame that are relevant to the video's subject matter. A user can touch different areas of the video frame (e.g., using a touchscreen), and when the user touches one of these relevant objects, the user's device can generate a haptic response (e.g., vibration) to provide feedback to the user that the user is touching the relevant object. In some embodiments, in this way, blind or visually impaired users can understand what is happening in the video without using a descriptive audio track.

[0014] Figure 1 Example system 100 is shown. (e.g.) Figure 1 As seen, system 100 includes computing device 104, network 106, and video server 108. Typically, computing device 104 generates tactile responses to touches on regions of interest in the video. In a particular embodiment, the tactile responses help blind or visually impaired user 102 understand the spatial relationships of content or events in the video.

[0015] User 102 uses computing device 104 to interact with other components of system 100. For example, computing device 104 may receive and present video from video server 108. Alternatively or additionally, the video may be stored in the memory of computing device 104 or loaded directly onto computing device 104 (e.g., via portable storage media such as optical discs or storage drives). When presenting video to user 102, computing device 104 may detect a touch on a region of interest in the video. Computing device 104 may then generate a haptic response to alert user 102 that the touch occurred on the region of interest. Figure 1As shown, computing device 104 includes a processor 114, a memory 116, and a touchscreen 118, which can be configured to perform the actions and functions of computing device 104 described herein. Computing device 104 also includes at least one of the following: (i) one or more layers 119, such as conductive or insulating layers disposed on, for example, the surface of touchscreen 118; (ii) an actuator 120; or (iii) a speaker 122. Some embodiments of computing device 104 may not include each of these components. For example, computing device 104 may include only one of the following: (i) one or more layers 119, (ii) an actuator 120, or (iii) a speaker 122. In another example, computing device 104 may include only two of the following: (i) one or more layers 119, (ii) an actuator 120, or (iii) a speaker 122. In yet another example, computing device 104 may include each of the following: (i) one or more layers 119, (ii) an actuator 120, and (iii) a speaker 122. As another example, different embodiments of the touchscreen 118 may include different types of layers 119.

[0016] Computing device 104 includes any suitable device for communicating with components of system 100 via network 106. By way of example and not limitation, computing device 104 may be a computer, laptop computer, wireless or cellular phone, e-notebook, personal digital assistant, tablet computer, or any other device capable of receiving, processing, storing information, or communicating information with other components of system 100. Computing device 104 may be a wearable device, such as a virtual reality or augmented reality headset, smartwatch, or smart glasses. Computing device 104 may be a video game system or console. Regarding touchscreen 118, touchscreen 118 may be physically integrated with computing device 104 or may be communicatively coupled to computing device 104 but physically separate from computing device 104. For example, in the case where computing device 104 is implemented as a smartphone, laptop computer, or tablet computer, touchscreen 118 will typically be integrated with computing device 104. In contrast, in the case where computing device 104 is implemented as a desktop computer, touchscreen 118 may take the form of a monitor separate from the computer tower-shaped computing device 104. The computing device 104 may also include a user interface, such as a microphone, keyboard, or other suitable terminal device available to the user 102. The computing device 104 may include a hardware processor, memory, or circuitry configured to perform any of the functions or actions of the computing device 104 described herein. For example, a software application designed using software code may be stored in memory and executed by the processor to perform the functions of the computing device 104.

[0017] Processor 114 is any electronic circuit, including but not limited to a microprocessor, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), and / or state machine, communicatively coupled to memory 116 and controlling the operation of computing device 104. Processor 114 may be 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. Processor 114 may include an arithmetic logic unit (ALU) for performing arithmetic and logical operations, processor registers for providing operands to the ALU and storing the results of ALU operations, and control units for fetching instructions from memory and executing them by directing the coordinated operation of the ALU, registers, and other components. Processor 114 may include other hardware that operates software to control and process information. Processor 114 executes software stored in memory to perform any of the functions described herein. Processor 114 controls the operation and management of computing device 104 by processing information (e.g., information received from video server 108, network 106, and memory 116). Processor 114 may be a programmable logic device, microcontroller, microprocessor, any suitable processing device, or any suitable combination thereof. Processor 114 is not limited to a single processing device and may contain multiple processing devices.

[0018] Memory 116 may permanently or temporarily store data, operating software, or other information of processor 114. Memory 116 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, memory 116 may include random access memory (RAM), read-only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage devices or combinations thereof. Software represents any suitable set of instructions, logic, or code contained in a computer-readable storage medium. For example, software may be embodied in memory 116, a disk, CD, or flash drive. In a particular embodiment, software may include an application program executable by processor 114 to perform one or more of the functions described herein.

[0019] Touchscreen 118 may include a display and touch detection circuitry. Typically, the touch detection circuitry may cover a portion of the display, allowing touchscreen 118 to present visual information (e.g., video) via the display and detect touches via the touch detection circuitry. Touchscreen 118 may present video 124 to user 102. Video 124 may be received from video server 108 (e.g., via streaming). In some embodiments, video 124 may already be stored within computing device 104 instead of being received from video server 108. When user 102 instructs computing device 104 to play video 124, computing device 104 may present the video on touchscreen 118. Video 124 includes sequentially played individual image frames 126. Each frame 126 may be a still image. Playing video 124 causes computing device 104 to sequentially play these frames 126 on touchscreen 118.

[0020] Computing device 104 can use saliency map 128 to identify one or more regions of interest 130 in one or more frames 126 of video 124. In a particular embodiment, video server 108 generates saliency map 128 and transmits it to computing device 104. For example, video server 108 may transmit saliency map 128 in a separate stream from the video 124 stream. As another example, video server 108 may transmit saliency map 128 in the same stream as video 124, which can provide more reliable synchronization. In some embodiments, computing device 104 may process video 124 and frames 126 to generate saliency map 128, which can reduce the amount of data that video server 108 stores, processes, and transmits. Typically, saliency map 128 identifies one or more regions of interest 130 in frames 126. Saliency map 128 can be generated using any suitable process, such as one or more processes described in U.S. Application No. 16 / 840,201, which is incorporated herein by reference. In some embodiments, a human operator may manually generate saliency map 128 by marking regions or objects of interest in frames 126 of video 124.

[0021] While computing device 104 is playing video 124, user 102 may touch touchscreen 118. When user 102 touches touchscreen 118, touchscreen 118 may detect or register touch 132. For example, touchscreen 118 may detect touch 132 occurring and record the coordinates of touch 132. The coordinates may indicate the area on touchscreen 118 where touch 132 is occurring. Computing device 104 may then determine whether a portion of the area on touchscreen 118 where touch 132 is occurring overlaps with a portion of region of interest 130 in frame 126 indicated by salience 128. If a portion of the area on touchscreen 118 where touch 132 is occurring overlaps with a portion of region of interest 130, computing device 104 may generate a response to alert user 102 that user 102 is touching region of interest 130. Computing device 104 may determine that the area on touchscreen 118 where touch 132 is occurring overlaps with region of interest 130 when computing device 104 determines that these areas partially overlap or one of these areas completely overlaps with the other. In other words, this disclosure considers regional overlap when the regions partially or completely overlap.

[0022] Computing device 104 may use actuator 120, touchscreen 118, or speaker 122 to generate a response to user 102. For example, computing device 104 may use actuator 120, speaker 122, or touchscreen 118 to generate a tactile response 134 that user 102 can feel when touching computing device 104. Actuator 120 may be any suitable actuator, such as a piezoelectric actuator or an electromagnetic actuator (e.g., a motor). Computing device 104 may activate actuator 120 to generate tactile response 134, which may be a mechanical vibration or pulse within computing device 104. Alternatively, or in addition to using actuator 120, computing device 104 may activate speaker 122 to generate tactile response 134. Speaker 122 may generate a low-frequency audio signal that causes mechanical vibration or pulsation within computing device 104. User 102 can feel mechanical pulsations or vibrations generated by actuator 120 or low-frequency audio signals (e.g., vibrations) from speaker 122 when touching computing device 104.

[0023] Alternatively, or in addition to activating actuator 120 or speaker 122, computing device 104 may use touchscreen 118 to generate haptic response 134. For example, touchscreen 118 may include one or more layers 119, such as conductive layers and thin insulating layers. Layers 119 may form the surface of touchscreen 118, and computing device 104 may transmit electrical signals through one or more layers 119 (e.g., through conductive layers) such that the electrical signals cause user 102 to experience a sensation when touching touchscreen 118. For example, when user 102 is touching touchscreen 118, the electrical signals may cause user 102 to perceive a sensation similar to touchscreen 118 changing from smooth to rough (also known as electrical vibration). Although the physical texture of touchscreen 118 does not change, the electrical signals cause user 102 to feel that the texture of touchscreen 118 is changing. As another example, computing device 104 may transmit electrical signals that actually change the physical texture of touchscreen 118 (compared to the previous example where the physical texture of touchscreen 118 does not change but user 102 feels a change in texture). Touchscreen 118 may include one or more layers 119 of a material whose physical texture is altered based on electrical signals. For example, touchscreen 118 may include a magnetorheological fluid (e.g., contained in layer 119) that alters shape (e.g., from round to sharp), texture (e.g., from smooth to bumpy or wavy), or hardness (e.g., from soft to hard) based on an electromagnet controlled by electrical signals. When the touchscreen 118 is touched, user 102 can feel these texture changes.

[0024] In some embodiments, computing device 104 may modify certain aspects of the haptic response 134 based on the region of interest 130 that user 102 is touching. For example, computing device 104 may modify the intensity or frequency of the haptic response 134 based on one or more of the size of an object in the region of interest 130, the curvature of the object, or the depth of the object in video frame 126 (e.g., whether the object is in the foreground, midground, or background region of frame 126). Computing device 104 may apply weights to one or more of these factors and modify the intensity or frequency of the haptic response 134 based on a weighted sum of these factors. As another example, computing device 104 may modify the intensity or frequency of the haptic response 134 based on the salience level of the region of interest 130 indicated by salience map 128 (e.g., the more salience, importance, or relevance the region of interest 130, the higher the intensity or frequency of the haptic response 134). In certain embodiments, by modifying the haptic response 134, computing device 104 may inform user 102 of different characteristics of the region of interest 130. For example, computing device 104 can inform user 102 of the size, curvature, or depth of objects in the region of interest. As another example, computing device 104 can inform user 102 of the salience, importance, or relevance of the region of interest 130.

[0025] In a particular embodiment, computing device 104 may use speaker 122 to generate an audio response 136 that is heard by user 102 when touch 132 occurs on a portion of touchscreen 118 that overlaps with a portion of region of interest 130. For example, speaker 122 may play an audio signal (different from the low-frequency signal generated for haptic response 134) to generate audio response 136. Audio response 136 provides additional audible signal to alert user 102 that touch 132 has occurred on region of interest 130. In some embodiments, computing device 104 generates both haptic response 134 and audio response 136.

[0026] Similar to haptic response 134, computing device 104 can modify certain aspects of audio response 136 based on the region of interest 130 that user 102 is touching. For example, computing device 104 can modify the volume or frequency of audio response 136 based on one or more of the size of an object in region of interest 130, the curvature of the object, or the depth of the object in video frame 126. Alternatively, computing device 104 can modify the volume or frequency of audio response 136 based on the salience level of region of interest 130 as indicated by salience figure 128 (e.g., the more salience, importance, or relevance of region of interest 130, the higher the volume or frequency of audio response 136). In a particular embodiment, by modifying audio response 136, computing device 104 can inform user 102 of different characteristics of region of interest 130.

[0027] Network 106 is any suitable network operable to facilitate communication between components of system 100. Network 106 may include any interconnected system capable of transmitting audio, video, signals, data, messages, or any combination thereof. Network 106 may include all or some of the following: Public Switched Telephone Network (PSTN), public or private data network, Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), local, regional, or global communications or computer network, such as the Internet, wired or wireless network, corporate intranet, or any other suitable communication link, including combinations thereof, operable to facilitate communication between components.

[0028] In some embodiments, video server 108 stores video 124 and transmits it to computing device 104. For example... Figure 1 As seen herein, video server 108 includes processor 110 and memory 112, which can be configured to perform any of the actions or functions of video server 108 described herein. In a particular embodiment, video server 108 can transmit video 124 to computing device 104 in a stream.

[0029] Processor 110 is any electronic circuit, including but not limited to a microprocessor, ASIC, ASIP, and / or state machine, communicatively coupled to memory 112 and controlling the operation of video server 108. Processor 110 may be 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. Processor 110 may include an ALU for performing arithmetic and logical operations, processor registers for providing operands to the ALU and storing the results of ALU operations, and a control unit for fetching instructions from memory and executing them by directing the coordinated operation of the ALU, registers, and other components. Processor 110 may include other hardware that operates software to control and process information. Processor 110 executes software stored in memory to perform any of the functions described herein. Processor 110 controls the operation and management of video server 108 by processing information (e.g., information received from computing device 104, network 106, and memory 112). Processor 110 may be a programmable logic device, microcontroller, microprocessor, any suitable processing device, or any suitable combination thereof. Processor 110 is not limited to a single processing device and may include multiple processing devices.

[0030] Memory 112 may permanently or temporarily store data, operating software, or other information of processor 110. Memory 112 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, memory 112 may include RAM, ROM, magnetic storage devices, optical storage devices, or any other suitable information storage devices or combinations thereof. Software represents any suitable set of instructions, logic, or code contained in a computer-readable storage medium. For example, software may be embodied in memory 112, a disk, CD, or flash drive. In a particular embodiment, software may include an application program executable by processor 110 to perform one or more of the functions described herein.

[0031] In some embodiments, video server 108 generates a saliency map 128 and streams it to computing device 104. Video server 108 processes frames 126 of video 124 using any suitable process to generate the saliency map 128. For example, video server 108 may use one or more processes described in U.S. Application No. 16 / 840,201, which is incorporated herein by reference.

[0032] Figure 2 yes Figure 1 The flowchart illustrates an example method 200 in system 100. Typically, computing device 104 executes method 200. In a particular embodiment, by executing method 200, computing device 104 generates a response that helps blind or visually impaired user 102 understand the content or events in video 124. While blocks of method 200 are shown... Figure 2The blocks in method 200 are presented in a specific order, but the blocks can be executed in any suitable order. For example, some blocks can be executed in a different order (e.g., block 204 can be executed before block 202). As another example, some blocks can be executed simultaneously (e.g., blocks 202 and 206 can be executed simultaneously).

[0033] In block 202, computing device 104 displays video 124, including video frame 126. For example, computing device 104 may display video 124 and video frame 126 on touchscreen 118. Computing device 104 may receive video 124 in a stream from video server 108.

[0034] In block 204, computing device 104 determines a region of interest 130 in video frame 126. Computing device 104 may use saliency map 128 to identify the region of interest 130. In some embodiments, computing device 104 may have already received saliency map 128 from video server 108. For example, video server 108 may have already transmitted saliency map 128 to computing device 104 in a separate stream from the video 124 stream. In some embodiments, computing device 104 may process video 124 or video frame 126 to generate saliency map 128. Typically, saliency map 128 indicates one or more regions of interest 130 in video frame 126. These regions of interest 130 may include one or more objects in video frame 126 that are identified as salient or relevant. Computing device 104 or video server 108 may use one or more processes described in U.S. Application No. 16 / 840,201 (which is incorporated herein by reference) to analyze video frame 126 or video 124 to identify these salient or relevant objects and determine the region of interest 130.

[0035] In block 206, computing device 104 detects a touch 132 on an area of ​​touchscreen 118 while displaying frame 126. Touchscreen 118 may include touch detection circuitry capable of detecting touch 132. Computing device 104 can detect touch 132 and determine the coordinates of touch 132. The coordinates may indicate the area of ​​touchscreen 118 where touch 132 occurred.

[0036] In block 208, computing device 104 determines whether the area of ​​touch 118 where touch 132 occurs overlaps with the region of interest 130 indicated by salience 128. In other words, computing device 104 determines whether user 102 is touching region of interest 130 in frame 126. If user 102 is not touching region of interest 130, computing device 104 can terminate method 200 without generating a response. If user 102 is touching region of interest 130, computing device 104 generates a response in block 210. For example, computing device 104 can generate a tactile response 134 that user 102 can feel. Computing device 104 can activate actuator 120 to generate a mechanical vibration or pulse that user 102 can feel when touching computing device 104. As another example, computing device 104 can activate speaker 122 to generate a low-frequency audio signal that produces a mechanical vibration or pulse that user 102 can feel when touching computing device 104. As another example, computing device 104 can transmit electrical signals via layer 119 of touchscreen 118, such that when user 102 touches layer 119, the electrical signals cause user 102 to perceive a sensation. This haptic response 134 alerts user 102 that user 102 is touching area of ​​interest 130, which helps user 102 understand the content or events within video 124. Alternatively or additionally, computing device 104 can generate an audio response 136 that can be heard by user 102. Computing device 104 can activate speaker 122 to generate an audio signal that can be heard by user 102 to alert user 102 that user 102 is touching area of ​​interest 130. Audio response 136 helps user 102 understand the content or events within video 124.

[0037] In some embodiments, computing device 104 may modify certain aspects of the haptic response 134 based on the region of interest 130 that user 102 is touching. For example, computing device 104 may modify the intensity or frequency of the haptic response based on the size of an object in the region of interest, the curvature of the object, or the depth of the object in video frame 126. Additionally or alternatively, computing device 104 may modify the intensity or frequency of the haptic response 134 based on the salience level of the region of interest 130 indicated by salience diagram 128 (e.g., the more salience, importance, or relevance the region of interest 130, the higher the intensity or frequency of the haptic response 134). In a particular embodiment, by modifying the haptic response 134, computing device 104 may inform user 102 of different characteristics of the region of interest 130.

[0038] Figure 3 yes Figure 1The flowchart illustrates an example method 300 in system 100. Computing device 104 or video server 108 can execute method 300. In a particular embodiment, by executing method 300, computing device 104 or video server 108 generates a saliency map 128, which can be used to help blind or visually impaired user 102 understand content or events within video 124. For example, in block 204 of method 200, saliency map 128 can be used by computing device 104 to identify a region of interest 130 in video frame 126. Then, when user 102 touches the region of interest 130, computing device 104 can generate a tactile response 134 in block 210 of method 300.

[0039] In block 302, computing device 104 or video server 108 detects an object of interest in video frame 126. The object of interest (e.g., a person, an important object, etc.) can be an object in video frame 126 that is related to or important to an event occurring in video frame 126. Computing device 104 or video server 108 can use any suitable process to detect the object of interest, such as one or more processes described in U.S. Application No. 16 / 840,201, which is incorporated herein by reference.

[0040] In block 304, computing device 104 or video server 108 determines the coordinates and characteristics of the object of interest. The coordinates indicate the position or location of the object of interest within frame 126. The characteristics of the object of interest may include its size, curvature, or depth within video frame 126. Computing device 104 may use one or more of these characteristics to generate appropriate haptic responses 134 and / or audio responses 136.

[0041] In block 306, computing device 104 or video server 108 generates a saliency map 128 indicating a region of interest 130 including an object of interest. For example, computing device 104 or video server 108 may use the coordinates of the object of interest to define the region of interest 130 in the saliency map 128. Additionally, computing device 104 or video server 108 may use characteristics of the object of interest to determine its relevance or importance. The saliency map 128 may include a saliency level indicating the importance or relevance of the region of interest 130, corresponding to the relevance or importance of the object of interest. In this way, computing device 104 or video server 108 generates the saliency map 128, which may later be used to determine when user 102 has touched the region of interest 130. In some embodiments, the saliency map 128 may indicate a saliency level within the region of interest 130. For example, certain portions of the region of interest 130 may be more saliency or more important than other portions of the region of interest 130. The saliency map 128 may indicate how saliency increases or decreases throughout the region of interest 130. When a user touches different parts of the region of interest 130, the computing device 104 can change its response based on the importance of that part as indicated by the salience diagram 128.

[0042] Figure 4-7 It shows Figure 1 Example operations in System 100. Typically, Figure 4-7 The operation of computing device 104 is shown when user 102 touches area of ​​interest 130.

[0043] Figure 4 It shows Figure 1 Example video frame 126 in system 100. Figure 4 As seen, computing device 104 includes a touchscreen 118 that displays video frame 126. Video frame 126 includes two people in a natural environment. Computing device 104 can display video frame 126 as part of playing video 124.

[0044] Figure 5 It shows Figure 1 Example saliency diagram 128 in system 100, which can be used Figure 3 Method 300 is generated. For example... Figure 5As seen, saliency map 128 identifies three regions of interest 130 in video frame 126. Regions of interest 130 are aligned with two people and a tree in video frame 126. In other words, the two people and the tree are objects of interest in video frame 126. As discussed previously, computing device 104 can receive saliency map 128 from video server 108, or computing device 104 can process video frame 126 to generate saliency map 130. Computing device 104 can use saliency map 128 to determine when user 102 has touched a region of interest 130 in video frame 126.

[0045] Figure 6 It shows Figure 1 An example of touch detection in System 100. For example... Figure 6 As seen, computing device 104 detects a touch 132 on an area of ​​touchscreen 118. Touch 132 does not occur on one of the people or trees in video frame 126 (i.e., touch 132 is outside / does not overlap with any region of interest 130). As a result, computing device 104 does not generate a response to touch 132, such as haptic response 134 or audio response 136.

[0046] Figure 7 It shows Figure 1 Example haptic response 134 in system 100. For example... Figure 7 As seen, computing device 104 detects touch 132 on touchscreen 118. Because touch 132 is on one of the people in video frame 126, computing device 104 determines from salience view 128 that touch 132 is on an area of ​​touchscreen 118 that overlaps with region of interest 130 in salience view 128. This region of interest 130 corresponds to one of the people in video frame 126. In response, computing device 104 generates haptic response 134. For example, computing device 104 may activate motor 120 or speaker 122 to induce mechanical vibrations or pulses that user 102 can feel when touching computing device 104. In this way, computing device 104 alerts user 102 that user 102 is touching region of interest 130 in video frame 126. In a particular embodiment, haptic response 134 helps blind or visually impaired user 102 understand the location of a salient element (i.e., one of the people) in video frame 126.

[0047] In a particular embodiment, computing device 104 can modify certain aspects of the haptic response 134 based on characteristics of the region of interest 130 being touched. For example, computing device 104 can modify the intensity or frequency of the haptic response 134 based on the size of the object in the region of interest 130, the curvature of the object in the region of interest 130, the depth of the object in the video frame 126, the salience level of the region of interest 130, or a combination thereof. The intensity or frequency of the haptic response 134 can also vary based on the relative importance of the touched object. For example, in Figure 4-7 In this context, people in the foreground may be more salient, important, or relevant than people in the background, thus a touch on a person in the foreground is associated with a higher intensity or frequency of tactile response 134 compared to a touch on a person in the background. In the same example, both people may be more important than the tree, such that a touch on a person in the foreground is associated with a higher intensity or frequency of tactile response 134 compared to a touch on a person in the background, and a touch on a person in the background is associated with a higher intensity or frequency of tactile response 134 compared to a touch on a tree. By altering these aspects of tactile response 134, computing device 104 can communicate one or more of these characteristics of region 130 of interest to blind or visually impaired user 102.

[0048] Various embodiments have been referenced in this disclosure. However, it should be understood that this disclosure is not limited to the embodiments specifically described. Rather, any combination of the following features and elements, whether or not associated with different embodiments, is contemplated to implement and practice the teachings provided herein. Furthermore, when elements of an embodiment are described in the form of "at least one of A and B," it will be understood that embodiments including only element A, only element B, and elements A and B are all contemplated. Moreover, while some embodiments may achieve advantages over other possible solutions or over the prior art, whether a particular advantage is achieved by a given embodiment does not limit this disclosure. Therefore, the aspects, features, embodiments, and advantages disclosed herein are illustrative only and should not be considered elements or limitations of the appended claims unless expressly stated in the claims. Similarly, references to "the invention" should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered elements or limitations of the appended claims unless expressly stated in the claims.

[0049] As those skilled in the art will understand, the embodiments described herein can be embodied as systems, methods, or computer program products. Therefore, embodiments can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which can be collectively referred to herein as “circuit,” “module,” or “system.” Furthermore, the embodiments described herein can take the form of computer program products embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0050] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination thereof.

[0051] Computer program code used to perform the operations of embodiments of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, Swift, Objective-C, C#, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0052] Aspects of this disclosure are described herein with reference to flowchart illustrations or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustration or block diagram, and combinations of blocks in the flowchart illustration or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in the blocks of the flowchart illustration or block diagram.

[0053] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of art, including instructions that implement the functions / actions specified in blocks of a flowchart description or block diagram.

[0054] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operational steps to be performed on the computer, other programmable apparatus or other equipment to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable data processing apparatus or other equipment, provide for implementing the functions / actions specified in the blocks of the flowchart description or block diagram.

[0055] The flowchart illustrations and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart illustrations or block diagrams may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may not appear in the order indicated in the figures. For example, depending on the functions involved, two blocks shown consecutively may actually execute substantially simultaneously, or the blocks may sometimes execute in reverse order or out of order. It will also be noted that each block in the block diagram or flowchart illustration, and combinations of blocks in the block diagram or flowchart illustration, may be implemented by a dedicated hardware-based system that performs the specified function or action, or a combination of dedicated hardware and computer instructions.

[0056] While the foregoing describes embodiments of this disclosure, other and further embodiments of this disclosure may be devised without departing from its essential scope, the scope of which is defined by the appended claims.

Claims

1. A method comprising: displaying, on a touchscreen, a video comprising a video frame; determining, based on a saliency map of the video frame, a region of interest in the video frame, the saliency map comprising a saliency level indicating an importance or relevance of one or more portions of the region of interest; detecting, while displaying the video frame, a touch on a region of the touchscreen; in response to determining that at least a portion of the region of the touchscreen overlaps at least a portion of the region of interest, generating a haptic response, the haptic response having an aspect determined based on the saliency map, wherein the aspect comprises at least one of an intensity or a frequency; and changing the aspect based on an importance or relevance indicated in the saliency map associated with the at least a portion of the region of interest.

2. The method of claim 1, wherein, The haptic response is generated using an actuator or a speaker.

3. The method of claim 1, further comprising: detecting an object of interest in the video frame; and and generating the saliency map such that the region of interest comprises the object of interest.

4. The method of claim 1, wherein, The intensity or the frequency is further changed based on one or more of a size of an object of interest in the video frame, a curvature of the object, or a depth of the object in the video frame.

5. The method of claim 1, wherein, The intensity is changed based on a saliency level in the saliency map.

6. The method of claim 1, further comprising: An audio response is generated in response to determining that the region of the touchscreen overlaps the region of interest.

7. The method of claim 1, wherein, The haptic response is generated by transmitting an electrical signal through a layer of the touchscreen.

8. An apparatus comprising: a touchscreen configured to display a video comprising a video frame; and a hardware processor communicatively coupled with the touchscreen, the hardware processor configured to: determine, based on a saliency map of the video frame, a region of interest in the video frame, the saliency map comprising a saliency level indicating an importance or relevance of one or more portions of the region of interest; detect, while displaying the video frame, a touch on a region of the touchscreen; in response to determining that at least a portion of the region of the touchscreen overlaps at least a portion of the region of interest, generate a haptic response, the haptic response having an aspect determined based on the saliency map, wherein the aspect comprises at least one of an intensity or a frequency; and change the aspect based on an importance or relevance indicated in the saliency map associated with the at least a portion of the region of interest. at least one of an actuator or a speaker, wherein the haptic response is generated using the at least one of the actuator or the speaker.

9. The apparatus of claim 8, further comprising: The hardware processor is further configured to:

10. The apparatus of claim 8, wherein, detect an object of interest in the video frame; and generate the saliency map such that the region of interest comprises the object of interest. The intensity or the frequency is further changed based on one or more of a size of an object of interest in the video frame, a curvature of the object, or a depth of the object in the video frame.

11. The apparatus of claim 8, wherein, The intensity is changed based on a saliency level in the saliency map.

12. The apparatus of claim 8, wherein, ​ 13. The apparatus of claim 8, further comprising: A speaker, wherein the hardware processor is further configured to generate an audio response using the speaker in response to determining that the region of the touchscreen overlaps with the region of interest.

14. The apparatus of claim 8, wherein, The haptic response is generated by transmitting an electrical signal through the touchscreen.

15. A system comprising: a server configured to transmit a video comprising video frames; and a computing device comprising: a touchscreen configured to display the video from the server; and a hardware processor communicatively coupled with the touchscreen, the hardware processor configured to: determine a region of interest in the video frames based on a saliency map of the video frames, the saliency map comprising a saliency level indicating an importance or relevance of one or more portions of the region of interest; detect a touch on a region of the touchscreen while displaying the video frames; and generate a haptic response in response to determining that at least a portion of the region of the touchscreen overlaps with at least a portion of the region of interest, the haptic response having an aspect determined based on the saliency map, wherein the aspect comprises at least one of an intensity or a frequency; and change the aspect based on an importance or relevance indicated in the saliency map associated with the at least a portion of the region of interest.

16. The system of claim 15, wherein, The computing device further comprises at least one of an actuator or a speaker, wherein the haptic response is generated using the at least one of the actuator or the speaker.

17. The system of claim 15, wherein, The intensity or the frequency is further changed based on one or more of a size of an object of interest in the video frames, a curvature of the object, or a depth of the object in the video frames.

18. The system of claim 15, wherein, The intensity is changed based on a saliency level in the saliency map.

19. The system of claim 15, wherein, The computing device further comprises a speaker, and wherein the hardware processor is further configured to generate an audio response using the speaker in response to determining that the region of the touchscreen overlaps with the region of interest.

20. The system of claim 15, wherein, The haptic response is generated by transmitting an electrical signal through the touchscreen. The haptic response is generated by transmitting an electrical signal through the touchscreen.

Images