Hemispherical cubemap projection format in imaging environments

Abstract

The present disclosure relates to a hemi-cube map projection format in an imaging environment. According to one embodiment, mechanisms for facilitating a hemi-cube map projection format imaging environment are described. The method of embodiments as described herein includes capturing, by a camera coupled to one or more processors, an image having image content, wherein the image content represented by the image is omnidirectional such that the image content, while representing less than a sphere, is mapped on a sphere; mapping the image to a cube representation based on six faces of the cube, wherein one or more of the six faces are classified as inactive regions such that the one or more faces remain unoccupied or partially occupied; and arranging the six faces in a compact representation based on the cube representation by avoiding inclusion of the inactive regions.

Description

[0001] This application is a divisional application of Chinese Invention Patent Application No. 201911207384.3, filed on November 29, 2019, entitled "Hemispherical Cube Mapping Projection Format in Imaging Environment".

[0002] Related applications

[0003] This application claims the benefit and priority of U.S. Provisional Application No. 62 / 787,156, filed on December 31, 2018, entitled “HEMISPHERE CUBE MAP PROJECTION FORMAT”, by Jill Boyce et al., the entire contents of which are incorporated herein by reference. Technical Field

[0004] The embodiments described herein relate generally to data processing, and more specifically to facilitating imaging environments in the form of hemisphere cube map projection. Background Technology

[0005] There is no known conventional technique for defining a cube map projection (CMP) format for hemispherical omnidirectional content. Such conventional techniques for cube map layout are often inefficient and prone to corruption. Summary of the Invention

[0006] Embodiments of this disclosure provide methods, apparatus, and computer-readable media for facilitating hemispherical cube mapping projection formats in digital imaging environments. The method includes capturing an image having image content by a camera coupled to one or more processors, wherein the image content represented by the image is omnidirectional such that the image content, although represented smaller than a sphere, is mapped onto a sphere; mapping the image to a cube representation based on the six faces of a cube, wherein one or more of the six faces are classified as inactive regions, such that one or more faces remain unoccupied or partially occupied; and arranging the six faces in a compact representation based on the cube representation by avoiding the inclusion of inactive regions. The apparatus includes corresponding means for performing the above-described method. The computer-readable medium includes instructions stored thereon that, when executed by one or more computing devices, cause the one or more computing devices to perform the above-described method. Attached Figure Description

[0007] The embodiments are illustrated in the accompanying drawings by way of example rather than limitation, and similar reference numerals in the drawings refer to similar elements.

[0008] Figure 1A computing device employing a forward projection mechanism is illustrated according to one embodiment.

[0009] Figure 2 A diagram is illustrated according to one embodiment. Figure 1 The forward projection mechanism and the reverse projection mechanism.

[0010] Figure 3A The illustration shows a traditional cube map layout.

[0011] Figure 3B The illustration shows a traditional CMP-based image.

[0012] Figure 3C The illustration shows a traditional image based on the equirectangular projection (ERP) format.

[0013] Figure 3D The image illustrates a traditional cropped ERP-based image.

[0014] Figure 3E The illustration shows a traditional image based on the semi-ERP format.

[0015] Figure 3F The illustration shows a traditional cropped CMP-based image with inactive areas.

[0016] Figure 4A A hemispherical CMP layout is illustrated according to one embodiment.

[0017] Figure 4B A two-dimensional hemispherical CMP projection with filling is illustrated according to one embodiment.

[0018] Figure 4C A two-dimensional half-ERP projection with filling is illustrated according to one embodiment.

[0019] Figure 4D A three-dimensional spherical view is illustrated according to one embodiment.

[0020] Figure 4E A hemispherical CMP format with weighted spherical peak signal-to-noise ratio is illustrated according to one embodiment.

[0021] Figure 5 The illustration depicts a computing device capable of supporting and implementing one or more embodiments.

[0022] Figure 6A A transaction sequence on a panoramic video system with a hemispherical cube map is illustrated according to one embodiment.

[0023] Figure 6B A method for forward projection is illustrated according to one embodiment.

[0024] Figure 6C A method for reverse projection is illustrated according to one embodiment. Detailed Implementation

[0025] Many specific details are set forth in the following description. However, the embodiments described herein can also be implemented without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0026] The embodiments provide novel techniques to facilitate hemispherical cubemap projection formats that represent 180°×180° hemispherical omnidirectional images or videos, such as those captured by a single fisheye lens, so that they can subsequently be efficiently encoded using existing image or video codecs. This projection format can be described in a supplemental enhancement information (SEI) message to be included in video coding standards, such as High Efficiency Video Coding (HEVC).

[0027] Figure 1 A computing device 100 employing a forward projection mechanism 110 is illustrated according to one embodiment. The computing device 100 represents a communication and data processing device, including (but not limited to) intelligent voice command devices, intelligent personal assistants, home / office automation systems, home appliances (e.g., washing machines, televisions, etc.), mobile devices (e.g., smartphones, tablets, etc.), gaming devices, handheld devices, wearable devices (e.g., smartwatches, smart bracelets, etc.), virtual reality (VR) devices, head-mounted displays (HMDs), Internet of Things (IoT) devices, laptop computers, desktop computers, server computers, set-top boxes (e.g., internet-based cable TV set-top boxes, etc.), GPS-based devices, automotive infotainment devices, etc.

[0028] In some embodiments, computing device 100 includes, works with, is embedded in, or assists any number and type of other intelligent devices, such as (but not limited to) autonomous machines or artificial intelligence agents, such as mechanical agents or machines, electronic agents or machines, virtual agents or machines, electromechanical agents or machines, etc. Examples of autonomous machines or artificial intelligence agents may include (but are not limited to) robots, autonomous vehicles (e.g., self-driving cars, autonomous aircraft, autonomous ships, etc.), autonomous devices (autonomous construction vehicles, autonomous medical devices, etc.), etc. Furthermore, "autonomous vehicles" is not limited to automobiles, but may include any number and type of autonomous machines, such as robots, autonomous devices, home autonomous devices, etc., and any one or more tasks or operations involving such autonomous machines may be referred to interchangeably with autonomous driving.

[0029] Additionally, for example, computing device 100 may include a computer platform housing integrated circuits (“ICs”), such as a system-on-a-chip (“SoC” or “SOC”), which integrates various hardware and / or software components of computing device 100 onto a single chip. For example, computing device 100 includes a data processing device having one or more processors, including (but not limited to) a central processing unit 112 and a graphics processing unit 114 coexisting on a common semiconductor package.

[0030] As shown in the figure, in one embodiment, computing device 100 may include any number and type of hardware and / or software components, such as (but not limited to) a graphics processing unit (“GPU” or simply “graphics processor”) 114, a graphics driver (also referred to as “GPU driver”, “graphics driver logic”, “driver logic”, user-mode driver (UMD), UMD, user-mode driver framework (UMDF), UMDF, or simply “driver”) 116, a central processing unit (“CPU” or simply “application processor”) 112, memory 104, network devices, drivers, etc., and one or more input / output (I / O) sources 108, such as a touchscreen, touch panel, touchpad, virtual or conventional keyboard, virtual or conventional mouse, ports, connectors, etc. Computing device 100 may include an operating system (OS) 106, which acts as an interface between the hardware and / or physical resources of computing device 100 and the user.

[0031] It should be understood that for some implementations, a system with fewer or more components than the example described above may be preferred. Therefore, depending on many factors, such as price constraints, performance requirements, technological improvements, or other circumstances, any configuration of computing device 100 may vary between implementations.

[0032] The embodiment may be implemented as any one or a combination of the following: one or more microchips or integrated circuits interconnected by a motherboard, hard-wired logic, software stored in a memory device and executed by a microprocessor, firmware, application-specific integrated circuit (ASIC), and / or field-programmable gate array (FPGA). Terms such as “logic,” “module,” “component,” “engine,” “circuit,” “element,” and “mechanism” may include, for example, software, hardware, firmware, and / or combinations thereof.

[0033] In one embodiment, as shown, the forward projection mechanism 110 may be hosted by a memory 104 that communicates with one or more I / O sources 108 of the computing device 100, such as a microphone, speaker, etc. In another embodiment, the forward projection mechanism 110 may be part of or hosted by an operating system 106. In yet another embodiment, the forward projection mechanism 110 may be hosted or assisted by a graphics driver 116. In yet another embodiment, the forward projection mechanism 110 may be hosted by or part of a graphics processing unit (“GPU” or simply “graphics processor”) 114 or the firmware of the graphics processor 114; for example, the forward projection mechanism 110 may be embedded in or implemented as part of the processing hardware of the graphics processor 114 in the form of a misuse evaluation component 130. Similarly, in another embodiment, the forward projection mechanism 110 may be housed in or be part of a central processing unit (“CPU” or simply “application processor”) 112; for example, the forward projection mechanism 110 may be embedded in or implemented as part of the processing hardware of the application processor 112 in the form of a misuse evaluation component 120.

[0034] For example, any element of the forward projection assembly 120, 130 and / or the forward projection mechanism 110 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, GPUs, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field-programmable logic devices (FPLDs).

[0035] This novel technology is contemplated not to be implemented in software or hardware, and as will be further described in this document, it can be applied and implemented in software, hardware, firmware, or any combination thereof. Therefore, embodiments are also contemplated not to be limited to a particular implementation or reception of the forward projection mechanism 110, and one or more portions or components of the forward projection mechanism 110 can be used or implemented as hardware, software, firmware, or any combination thereof. Furthermore, for the purposes of this document, the phrase “communicate with” (including variations thereof) encompasses direct communication and / or indirect communication via one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or continuous communication, but also includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-off events.

[0036] The computing device 100 may accommodate one or more network interface devices to provide access to a network, such as a LAN, wide area network (WAN), metropolitan area network (MAN), personal area network (PAN), Bluetooth, cloud network, mobile network (e.g., 3G, 4G, etc.), intranet, Internet, etc. The network interface may include, for example, a wireless network interface with an antenna, wherein the antenna may represent one or more antennas. The network interface may also include, for example, a wired network interface for communicating with remote devices via a network cable, such as an Ethernet cable, coaxial cable, optical fiber, serial cable, or parallel cable.

[0037] Embodiments may be provided, for example, as a computer program product, which may include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines such as a computer, data processing machine, data processing apparatus, computer network, or other electronic device, cause the one or more machines to perform the operations according to the embodiments described herein. (See reference...) Figure 1 The machine may include one or more processors, such as a CPU, GPU, etc. Machine-readable media may include, but are not limited to, floppy disks, optical disks, compact disc-read-only memory (CD-ROM), magneto-optical disks, ROMs, random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0038] For example, when reading any device, method, or system claim of this patent that covers a purely software and / or firmware implementation, at least one element of the forward projection assembly 120, 130, and / or the forward projection mechanism 110 can be explicitly defined as including a non-transitory computer-readable storage device or disk containing the software and / or firmware, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disc, etc.

[0039] Additionally, one or more elements of the forward projection components 120, 130 and / or the forward projection mechanism 110 may be downloaded as a computer program product, wherein the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) through one or more data signals embodied in and / or modulated by a carrier or other propagation medium.

[0040] Throughout this document, the term "user" may be used interchangeably with "viewer," "observer," "speaker," "person," "individual," "end user," "developer," "programmer," "administrator," and so on. For example, in some cases, "user" may refer to an end user, such as a consumer accessing a client computing device, while in others, it may include a developer, programmer, system administrator, and so on, accessing a workstation that acts as a client computing device. Note that throughout this document, terms like "graphics domain" may be used interchangeably with "graphics processing unit," "graphics processor," or simply "GPU"; similarly, "CPU domain" or "host domain" may be used interchangeably with "computer processing unit," "application processor," or simply "CPU."

[0041] It should be noted that terms such as "node," "computing node," "server," "server device," "cloud computer," "cloud server," "cloud server computer," "machine," "host machine," "device," "computing device," "computer," and "computing system" are used interchangeably throughout this document. Also note that terms such as "application," "software application," "program," "software program," "package," and "software package" are used interchangeably throughout this document.

[0042] Additionally, throughout this document, terms such as “request,” “query,” “job,” “work,” “work project,” and “workload” are used interchangeably. Similarly, “application” or “agent” can refer to or include computer programs, software applications, games, workstation applications, etc., provided through an application programming interface (API), where the API is, for example, a free rendering API, such as the Open Graphics Library (OPL). ), 11. 12 etc., where “assignment” can be referred to interchangeably with “work unit” or “draw tool”, and “application” can be referred to interchangeably with “workflow” or simply “agent”.

[0043] For example, a workload, such as a 3D game workload, may include and issue any number and type of "frames," where each frame may represent an image (e.g., a sailboat, a face). Additionally, each frame may include and provide any number and type of work units, where each work unit may represent a portion of the image (e.g., a sailboat, a face) represented by its corresponding frame (e.g., the mast of a sailboat, the forehead of a face). However, for consistency, throughout this document, each item is referred to by a single term (e.g., "dispatch," "agent," etc.).

[0044] In some embodiments, terms such as “display screen” and “display surface” may be used interchangeably to refer to the visible portion of a display device, while the remainder of the display device may be embedded in a computing device, such as a smartphone, wearable device, etc. It is contemplated and noted that the embodiments are not limited to any particular computing device, software application, hardware component, display device, display screen or surface, protocol, standard, etc. For example, the embodiments can be applied to and used with any number and type of real-time applications on any number and type of computers, such as desktop computers, laptop computers, tablet computers, smartphones, head-mounted displays, and other wearable devices, etc. Furthermore, for example, this novel technology can be used to render scenes for high performance, ranging from simple scenes such as desktop environments to complex scenes such as 3D games, augmented reality applications, etc.

[0045] Figure 2 A diagram is illustrated according to one embodiment. Figure 1 The forward projection mechanism 110 and the reverse projection mechanism 260 are described below. For the sake of brevity, the details will not be repeated or discussed further. Figure 1 Many details are discussed. In one embodiment, the forward projection mechanism 110 may include any number and type of elements or components, such as (but not limited to): detection and monitoring logic 201; initiation logic 203; mapping logic 205; arrangement logic 207; communication / compatibility logic 209; and filling logic 211. The computing device 100 also accommodates a video encoder 213 and a user interface 219.

[0046] In the illustrated embodiment, computing device 100 represents a server computing device (also referred to as a "server device" or simply a "server") that is coupled to or communicates with one or more client computing devices, such as client computing devices (also referred to as "client devices" or simply "clients") that house the reverse projection mechanism 260. In one embodiment, the reverse projection mechanism 260 may include one or more elements or components, such as (but not limited to): cropping / bending logic 261; projection logic 263; viewport generation logic 265; format creation logic 267; position selection and display logic 268; and communication logic 269. Computing device 250 also houses a video decoder 273, a user interface 275, and a display device 277.

[0047] Similar to the forward projection mechanism 110 at computing device 100, the reverse projection mechanism 260 at computing device 250 may be hosted as software, hardware, firmware, or any combination thereof. For example, the reverse projection mechanism 260 may be hosted as instructions by memory at computing device 250, and / or hosted as one or more hardware components by or embedded in one or more processors at computing device 250. It is contemplated and noted that the embodiments are not limited to any particular implementation.

[0048] The computing device 100 also includes a user interface 219 (e.g., a graphical user interface (GUI), a web browser, a cloud-based platform user interface, a software application-based user interface, other user or application programming interfaces (APIs), etc.). The computing device 100 may also include one or more I / O sources 108 having one or more input components 231, such as one or more cameras 242 (e.g., ...). RealSense TM Camera), (one or more) microphones 241, sensors, detectors, keyboards, mice, etc., and (one or more) output components 233, such as (one or more) display devices or simply displays 244 (e.g., integrated displays, tensor displays, projection screens, display screens, etc.), (one or more) speaker devices or simply speakers, etc.

[0049] The computing device 100 is also illustrated to be able to access one or more databases 225 and / or one or more other computing devices and / or communicate with them via one or more communication media 230 (e.g., networks such as a neighborhood network, cloud network, intranet, Internet, etc.).

[0050] In some embodiments, database(s) 225 may include one or more storage media or devices, repositories, data sources, etc., containing any amount and type of information, such as data, metadata, etc., relating to any number and type of applications, such as data and / or metadata relating to one or more users, physical locations or regions, applicable laws, policies and / or regulations, user preferences and / or profiles, security and / or authentication data, details of history and / or preferences, etc.

[0051] As previously described, computing device 100 may accommodate one or more I / O sources 108 including one or more input components 231 and one or more output components 233. In one embodiment, the one or more input components 231 may include a sensor array, including but not limited to one or more microphones 241 (e.g., ultrasonic microphones), one or more cameras 242 (e.g., two-dimensional (2D) cameras, three-dimensional (3D) cameras, infrared (IR) cameras, depth-sensing cameras, etc.), capacitors, radio components, radar components, scanners and / or accelerometers, etc. Similarly, the one or more output components 233 may include any number and type of one or more display devices 244, projectors, light-emitting diodes (LEDs), one or more speakers 243, and / or vibration motors, etc.

[0052] As previously mentioned, terms such as “logic,” “module,” “component,” “engine,” “circuit,” “element,” and “mechanism” can include, for example, software, hardware, firmware, and / or any combination thereof. For instance, logic itself can be or can include circuitry at or associated with one or more devices, such as… Figure 1 The forward projection components 130 and / or 120, respectively hosted by application processor 112 and / or graphics processor 114, must facilitate or execute corresponding logic to perform certain tasks. Similarly, the reverse projection mechanism 260 may be hosted as one or more reverse projection components by one or more applications and / or graphics processors at computing device 250.

[0053] The embodiments provide novel techniques facilitated by forward projection mechanism 110 and / or reverse projection mechanism 260 for projecting hemispherical cubemap projection format to represent hemispherical 180°×180° omnidirectional images or videos, such as those captured by a single fisheye lens. For example, as referenced Figure 4AAs shown, this novel hemisphere cube map format (HCMP) 400 includes a full cube face 401 representing a hemisphere and four half cube faces 403A, 403B, 405A, and 405B. The full face 401 and the half faces 403A, 403B, 405A, and 405B are arranged in a different, more compact layout compared to existing CMP formats to reduce image size by eliminating inactive areas. In other words, this novel technique provides a better and more efficient way to deliver results. Furthermore, the proposed novel HCMP layout can be used with any cube-based projection format, as well as a hemisphere version of the Equi-Angular Cube map (EAC) format, where this novel format is referred to as the hemisphere EAC (HEAC) format. In one embodiment, these novel HCMP and HEAC formats can be added to software, such as 360Lib software.

[0054] For example, 360° × 180° omnidirectional content can be captured by using two fisheye camera lenses back-to-back and then stitching the content together. A single fisheye camera lens can capture 180° × 180° of content, representing a single hemisphere rather than a complete sphere. The fisheye camera is envisioned as an example, and the embodiments are not necessarily limited thereto. Additionally, for example, one or more cameras mentioned throughout this document may be included in camera(s)242, wherein camera(s)242 may be part of or embedded in computing device 100, or may be placed independently, such as on a street corner or in a room, and communicatively coupled to computing device 100 via one or more communication media 230.

[0055] Some content used in the MPEG 3DoF+ project within MPEG-I Visual is 180°×180° content in ERP format, such as the Technicolor Hijack sequence. The Omnidirectional Media Format (OMAF) defines the fisheye format, and the Joint Collaborative Team on Video Coding (JCT-VC) developed the Fisheye Projection Format Supplemental Enhancement Information (SEI) message, which enables encoding fisheye content in the original format in which it was captured. See the Joint Video Experts Team (JVET), 13th meeting, Marrakech, MA, January 9-18, 2019, ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC29 / WG 11. However, due to image warping, the fisheye format is not expected to be a bitrate-efficient format for video coding. Currently, there are no technologies (e.g., algorithms) available for implementing the fisheye projection equation, which JCT-VC is considering.

[0056] Traditionally, such as regarding Figure 3A As shown, it indicates based on Figure 3B The image 310 (e.g., from a Trolley sequence) is a CMP format layout 300 in 360Lib, showing various positions of different faces of a cube. As shown in the figure... Figure 3A The traditional CMP format layout 30 shows six faces, including front 301, back 302, left 303, right 304, top 305 and bottom 306. Figure 3C Image 320 illustrates another traditional ERP-based version of the Trolley sequence. Figure 3D The illustration shows a 180°×180° crop of the central content of a traditional ERP-based image 330, with black backgrounds 331A and 331B shown on either side. Figure 3E The image 340 is shown as a traditional semi-ERP format image where only the active area is preserved, as is used in the 3DoF+TechnicolorHijack sequence. Figure 3F The illustration shows a trolley image 350 cropped in traditional CMP format 180°×180°, indicating inactive or unused areas 351A, 351B, and 315C that cannot be directly cropped to reduce higher frame sizes, pixel rates, levels, etc.

[0057] The embodiment provides a novel technique in which a 180°×180° hemisphere can be projected onto half of a cube, wherein the active content is on a full face, such as the front 401, and four half faces, such as the left half 403A, right half 405A, lower half 405A, and upper half 405B, as referenced. Figure 4A As shown.

[0058] In one embodiment, at computing device 100, images and / or video (a sequence of images) are captured by one or more cameras 242, and detection and monitoring logic 201 continuously detects and monitors each image and its active and inactive regions. Upon detection of an active or inactive portion or region of an image, initiation logic 203 is triggered to initiate an initiation using any 180°×180° format of the image, wherein this initial format may include fisheye format, ERP format, etc. Following this initiation, mapping logic 205 is subsequently triggered to map the various portions of the image to form a single complete surface, e.g. Figure 4A The complete face 401, and multiple half faces, such as Figure 4A The four half-faces are 403A, 403B, 405A, and 405B.

[0059] In the proposed novel HCMP format, for example Figure 4A The HCMP format 400, and the layout logic 207 are subsequently used, for example, when using the CMP format, to arrange this full face 401 and half faces 403A-405B into a compact area to avoid any inactive areas. Additionally, in one embodiment, neighbor continuity within the hemisphere is maintained throughout the layout, where a single continuous area spans the left half face 403A, the front face 401, and the right half face 403B, which is facilitated by the layout logic 207, as referenced. Figure 4A As shown. Similarly, the upper half 405B and the lower half 405A are rotated and arranged on either side of a single continuous region, facilitated by arrangement logic 207, as... Figure 4A As shown.

[0060] Additionally, in one embodiment, filling logic 211 is triggered to fill objects such as... Figure 4A HCMP format 400 and similar formats add active padding by copying samples from neighbors on the sphere, such as padding sources A 409A and B 409B shown above and below the level of the previous 401, respectively, and respectively in Figure 4A The fill destinations A 407A and B 407B are shown vertically to the left of the left half 403A and the right of the right half 403B. Additionally, as facilitated by fill logic 211, active fill can be added at discontinuous boundaries copied (and rotated) from any corresponding neighboring location in the image.

[0061] Please refer to the following: Figure 4B and 4C Images 410 and 420 further illustrate, Figure 4A The filling destinations A 407A and B 407B and the filling sources A 409A and B 409B are shown as corresponding to respectively Figure 4B The filled destinations A, B, and C of image 410 (e.g., a two-dimensional (2D) HCMP projection image) are 407A and 407B. Figure 4C The filling sources A 409A and B 409B of the image 420 (e.g., a 2D half-ERP or HERP projection image). Figure 4D The diagram shows the button. Figure 4A The HCMP format 400 is further projected onto an image represented by a sphere 430. In one embodiment, the proposed novel layout is also applicable to other cube-based formats, such as EAC, HEC, etc.

[0062] Continuing forward to the projection mechanism 110, this novel format arrangement is then received at the video encoder 213, which encodes the relevant data and, facilitated by communication / compatibility logic 209 and / or communication logic 269, transmits the encoded data to the video decoder 273 at the computing device 250 via one or more communication media 230 such as a neighborhood network, cloud network, Internet, etc.

[0063] In one embodiment, video decoder 273 decodes the encoded data and provides it to reverse projection mechanism 260 for performing reverse projection and other related processes. For example, upon receiving the decoded data at reverse projection mechanism 260, clipping / bending logic 261 is triggered to clip or blend the fill area of ​​the novel format layout. In one embodiment, projection logic 263 is subsequently triggered to project the mapping of the novel format layout onto a hemisphere, for example, projecting the complete front along with the four hemispheres onto the hemisphere.

[0064] Once projection is complete, in one embodiment, viewport generation logic 265 can be used to generate a viewport for a user to view the image using display 277, where the viewport may be smaller than a full 180° × 180° hemisphere. Similarly, in another embodiment, format creation logic 267 is used to create a 180° × 180° hemisphere format within fisheye or ERP format content. As previously described, this image is then displayed to the user using display device 277 and facilitated by user interface 275.

[0065] Referring back to one or more input components 231, these may include any number and type of microphones 241, such as multiple microphones or microphone arrays, such as ultrasonic microphones, dynamic microphones, fiber optic microphones, laser microphones, etc. One or more of the microphones 241 are envisioned as acting as one or more input devices for accepting or receiving audio input (e.g., human speech) into computing device 100 and converting this audio or sound into electrical signals. Similarly, one or more cameras 242 are envisioned as acting as one or more input devices for detecting and capturing images and / or video of scenes, objects, etc., and providing the captured data as video input to computing device 100.

[0066] For example, as shown, input component 231 may include any number and type of microphones 241, such as multiple microphones or microphone arrays, such as ultrasonic microphones, dynamic microphones, fiber optic microphones, laser microphones, etc. It is envisioned that one or more of the microphones 241 act as one or more input devices for accepting or receiving audio input (e.g., human speech) into computing device 100 and converting this audio or sound into electrical signals. Similarly, it is envisioned that one or more cameras 242 act as one or more input devices for detecting and capturing images and / or videos of scenes, objects, etc., and providing the captured data as video input to computing device 100.

[0067] The JVET 360CTC sequence is cropped in the center to create a 180° × 180° sequence, as shown in the reference. Figure 3D As shown, and is considered the original sequence. For example, a sequence with a resolution of 8192×4096 is cropped to a resolution of 4096×4096, where all other stages in JVET 360CTC are followed except for halving the horizontal resolution, and HM 16.15 / 360Lib-3.0 and VTM 3.0 are used. Additionally, conditions for JVET-K1012-v1 are adapted for hemispherical formats; for example, for ERP, a face size of 2216×2216 is used, while for HCMP / HEAC, the face size is 1280×1280. Furthermore, HCMP uses an 8-pixel-wide padding for each face, so the total width is calculated as follows: 3*1280 (face) + 2*8 (padding) = 3856 pixels.

[0068] Additionally, the 360Lib software has been modified to support 180°×180° content, with support added for the proposed HCMP and HEAC formats. For example, the novel projection format is based on or provides the following: conversion to and / or from a 360° normalized sphere, 2DYUV read and / or write capabilities, 360Lib profile support extended to support half-face projection, source field-plate (FP) structural parameters, encoded FP structural parameters, and padding control defined by codingPCMP, input PCMP, etc. Furthermore, regarding the weighted spherical (WS) peak signal-to-noise ratio (PSNR), Figure 4E The diagram illustrates the WS-PSNR weights for HDMP format 440.

[0069] fisheye comparison

[0070] This novel HEAC format is compared with the original encoded fisheye format. For four simultaneously cropped ERP sequences in the sequence, the fisheye format is converted, for example, using Hugin open-source projection, intermediate conversion to red, green, and blue (RGB), and a resolution of 2048×2048. Traditional techniques do not provide support for the fisheye format in the 360Lib software.

[0071] Table 5 illustrates that the HEAC format saves between 68.6% and 81.1% of bit rate compared to the traditional fisheye format, and this is due to the introduction of multiple intermediate conversion steps with some coding overhead.

[0072] Table 5 HEAC and fisheye

[0073]

[0074]

[0075] The embodiments provide the addition of HCMP and HEAC to 360Lib and utilize cubemap-based formats, such as EAC, as shown above, to offer a significant bitrate reduction relative to ERP formats for 360°×180° spherical content. This novel technique provides similar benefits for 180°×180° spherical content, such as when images and / or videos are captured using a single fisheye lens camera (e.g., one or more cameras 242). Furthermore, this proposed novel layout requires smaller image sizes than when using existing CMP or EAC formats, enabling the encoding of hemispherical video sequences at lower levels.

[0076] Referring back to I / O source 108, embodiments are envisioned to include, but are not limited to, any number or type of microphones 241, cameras 242, speakers 243, displays 244, etc., for data capture or presentation. For example, as facilitated by detection and monitoring logic 201, one or more of the microphones 241 can be used to simultaneously detect speech or sound from a user (e.g., a speaker). Similarly, as facilitated by detection and monitoring logic 201, one or more of the cameras 242 can be used to capture images or videos of a geographic location (whether indoors or outdoors) and its associated content (e.g., furniture, electronic devices, people, animations, trees, mountains, etc.) and form a stream of images or videos.

[0077] Similarly, as shown, output component 233 may include any number and type of loudspeakers or loudspeaker devices 243 to act as output devices for outputting or giving output audio from computing device 100 for any number or type of reason (e.g., human listening or consumption). For example, loudspeakers 243 operate in contrast to microphones 241, wherein loudspeakers 243 convert electrical signals into sound.

[0078] Additionally, input components 231 may include any number or type of camera, such as depth-sensing cameras or capture devices (e.g., RealSense TM Depth-sensing cameras are known to be used to capture still and / or video red-green-blue (RGB) and / or RGB-depth (RGB-D) images for media (e.g., personal media). Such images with depth information have been effectively used for a variety of computer vision and computational photography effects, such as (but not limited to) scene understanding, refocusing, composition, moving pictures, and so on. Similarly, for example, displays may include any number and type of displays, such as integrated displays, tensor displays, stereoscopic displays, and so on, including (but not limited to) embedded or connected display screens, display devices, projectors, and so on.

[0079] Input components 231 may also include one or more of the following: vibration components, tactile components, conductivity elements, biometric sensors, chemical detectors, signal detectors, electroencephalography, functional near-infrared spectroscopy, wave detectors, force sensors (e.g., accelerometers), illuminators, eye-tracking or gaze-tracking systems, head-tracking systems, etc., which can be used to capture any amount and type of visual data, such as images (e.g., photographs, videos, movies, audio / video streams, etc.), and non-visual data, such as audio streams or signals (e.g., sound, noise, vibration, ultrasound, etc.), radio waves (e.g., wireless signals, such as wireless signals with data, metadata, symbols, etc.), chemical changes or properties (e.g., humidity, body temperature, etc.), biometric readings (e.g., fingerprints, etc.), brain waves, brain circulation, environmental / weather conditions, maps, etc. It is envisioned that "sensor" and "detector" may be used interchangeably throughout this document. It is also envisioned that one or more input components 231 may also include one or more supporting or supplementary devices for data capture and / or sensing, such as illuminators (e.g., IR illuminators), lamps, generators, sound blockers, etc.

[0080] It is also envisioned that in one embodiment, input component 231 may include any number and type of context sensors (e.g., linear accelerometers) for sensing or detecting any number and type of context (e.g., estimating a horizontal line, linear acceleration, etc., in relation to a mobile computing device, etc.). For example, input component 231 may include any number and type of sensors, such as (but not limited to): accelerometers (e.g., linear accelerometers used to measure linear acceleration, etc.); inertial devices (e.g., inertial accelerometers, inertial gyroscopes, micro-electro-mechanical systems (MEMS) gyroscopes, inertial navigators, etc.); and gravitational gradiometers to study and measure changes in gravitational acceleration due to gravity, etc.

[0081] Additionally, for example, input component 231 may include (but is not limited to): audio / visual devices (e.g., cameras, microphones, speakers, etc.); context-aware sensors (e.g., temperature sensors, facial expression and feature measurement sensors working with one or more cameras of the audio / visual devices, environmental sensors (e.g., for sensing background color, light, etc.); biometric sensors (e.g., for detecting fingerprints, etc.), calendar maintenance and reading devices), etc.); global positioning system (GPS) sensors; resource requesters; and / or TEE logic. TEE logic may be used alone or may be part of resource requesters and / or I / O subsystems, etc. Input component 231 may also include speech recognition devices, photo recognition devices, facial and other body recognition components, speech-to-text conversion components, etc.

[0082] Similarly, output component 233 may include a dynamic haptic touchscreen having a haptic effector as an example of presenting a visualization of touch, wherein an embodiment of such a dynamic haptic touchscreen may be an ultrasonic generator that can transmit signals in space that, when reaching, for example, a human finger, can evoke a tactile sensation or similar feeling on the finger. Additionally, for example, in one embodiment, output component 233 may include (but is not limited to) one or more of the following: a light source, a display device and / or screen, an audio speaker, a haptic component, a conductive element, a bone conduction speaker, an olfactory or odor visual and / or non-visual presentation device, a haptic or touch visual and / or non-visual presentation device, an animation display device, a biometric display device, an X-ray display device, a high-resolution display, a high dynamic range display, a multi-view display, and a head-mounted display (HMD) for at least one of virtual reality (VR) and augmented reality (AR), etc.

[0083] The embodiments are envisioned to be limited to any number or type of use case scenarios, architectural arrangements, or component setups; however, for the sake of brevity and clarity, illustrations and descriptions are provided and discussed throughout this document for illustrative purposes, but the embodiments are not limited thereto. Furthermore, throughout this document, “user” may refer to someone who has access to one or more computing devices (e.g., computing device 100) and may be used interchangeably with “person,” “individual,” “human,” “he,” “she,” “child,” “adult,” “viewer,” “player,” “gamer,” “developer,” “programmer,” and so on.

[0084] Communication / compatibility logic 209 can be used to facilitate dynamic communication and compatibility between various components, networks, (one or more) databases 225 and / or (one or more) communication media 230, etc., and any number and type of the following: other computing devices 250A, 250B, 250C, 260A, 260B, 260N (e.g., wearable computing devices, mobile computing devices, desktop computers, server computing devices, etc.), processing devices (e.g., central processing unit (CPU), graphics processing unit). Units, GPUs, etc.), capturing / sensing components (e.g., non-visual data sensors / detectors, such as audio sensors, olfactory sensors, tactile sensors, signal sensors, vibration sensors, chemical detectors, radio wave detectors, force sensors, weather / temperature sensors, body / biometric sensors, scanners, etc., and visual data sensors / detectors, such as cameras, etc.), user / context-aware components and / or identification / verification sensors / devices (e.g., biometric sensors / detectors, scanners, etc.), memory or storage devices, data sources and / or (one or more) databases (e.g., data storage devices, hard drives, solid-state drives, hard disks, memory cards or devices, memory circuits, etc.), (one or more) networks (e.g., cloud networks, the Internet, the Internet of Things, intranets, cellular networks, proximity networks, such as Bluetooth, Bluetooth Low Energy (BLE), Bluetooth Smart, Wi-Fi proximity, RFID, near-field communication, body area networks, etc.), wireless or wired communications and related protocols (e.g., WiMAX, Ethernet, etc.), connectivity and location management technologies, software applications / websites (e.g., social and / or business networking websites, business applications, games and other entertainment applications, etc.), programming languages, etc., while ensuring compatibility with changing technologies, parameters, protocols, standards, etc.

[0085] Throughout this document, terms such as “logic,” “component,” “module,” “framework,” “engine,” “tool,” and “circuit” are used interchangeably with, for example, software, hardware, firmware, and / or any combination thereof, and may include software, hardware, firmware, and / or any combination thereof. In one example, “logic” may refer to or may include a software component that works with one or more of the operating system, graphics driver, etc., of a computing device (e.g., computing device 100). In another example, “logic” may refer to or may include a hardware component that can be physically installed or installed as part of one or more system hardware elements of the computing device (e.g., computing device 100), such as an application processor, graphics processor, etc. In yet another example, “logic” may refer to or may include a firmware component that can be part of the system firmware of the computing device (e.g., computing device 100), such as the firmware of an application processor or graphics processor, etc.

[0086] Additionally, any use of specific trademarks, words, terms, phrases, names, and / or acronyms such as the following should not be construed as limiting the embodiments to software or devices bearing that label in products or literature outside this document: “hemisphere”, “cube mapping projection”, “hemisphere-cube mapping projection format”, “HCMP”, “equivalent rectangular projection”, “HERP”, “front”, “half-face”, “fill”, “projection mapping”, “forward projection”, “backward projection”, “viewport generation”, “backward projection”, “format creation”, “video encoding”, “video decoding”, “fisheye camera”, “fisheye format”, “mapping”, “layout”, “fill”, “cropping”, “blending”, “projection”, “creation”, “generation”, “depth”, “pixel depth”, “creation”, “training”, “inference”, “classification”, “estimation”, “RealSense” TM Camera, Real-time, Automatic, Dynamic, User Interface, Camera, Sensor, Microphone, Display Screen, Speaker, Verification, Authentication, Privacy, User, User Profile, User Preferences, Transmitter, Receiver, Personal Device, Smart Device, Mobile Computer, Wearable Device, IoT Device, Proximity Network, Cloud Network, Server Computer, etc.

[0087] It is envisioned that any number and type of components can be added. Figure 1 One or more of the forward projection mechanism 110 and / or forward projection components 120, 130 and / or Figure 2The reverse projection mechanism 260 (and / or one or more reverse projection components) and / or its removal facilitate various embodiments including the addition, removal, and / or enhancement of certain features. For simplicity, clarity, and ease of understanding... Figure 1 One or more of the forward projection mechanism 110 and / or forward projection components 120, 130 and / or Figure 2 The reverse projection mechanism 260 (and / or one or more reverse projection components) includes many standard and / or known components, such as those found in computing devices, which are not shown or discussed herein. The embodiments described herein are envisioned to be limited to any technology, topology, system, architecture, and / or standard and are dynamic enough to adopt and adapt to any future changes.

[0088] Figure 5 The illustration depicts a computing device 500 capable of supporting and implementing one or more embodiments. The illustrated computing device 500 can be connected to... Figure 2 The computing devices 100 and 250 are identical or similar. The computing device 500 houses a system board 502. Board 502 may include several components, including but not limited to a processor 504 and at least one communication packet 506. The communication packet is coupled to one or more antennas 516. The processor 504 is physically and electrically coupled to board 502.

[0089] Depending on its application, computing device 500 may include other components that may or may not be physically and electrically coupled to board 502. These other components include, but are not limited to, volatile memory (e.g., DRAM) 508, non-volatile memory (e.g., ROM) 509, flash memory (not shown), graphics processor 512, digital signal processor (not shown), encryption processor (not shown), chipset 514, antenna 516, display 518 (e.g., touchscreen display), touchscreen controller 520, battery 522, audio codec (not shown), video codec (not shown), power amplifier 524, global positioning system (GPS) device 526, compass 528, accelerometer (not shown), gyroscope (not shown), speaker 530, camera 532, microphone array 534, and mass storage devices (e.g., hard disk drive 510), CD (not shown), DVD (not shown), etc. These components may be connected to system board 502, mounted to system board, or combined with any other components.

[0090] Communication packet 506 enables wireless and / or wired communication for the transmission of data to and from computing device 500. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, technologies, communication channels, etc., that can transmit data via a non-solid-state medium using modulated electromagnetic radiation. This term does not imply that the associated devices do not contain any wiring, although in some embodiments they may indeed not. Communication packet 506 may implement any of several wireless or wired standards or protocols, including but not limited to Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, Long Term Evolution (LTE), Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, Bluetooth, Ethernet, and their derivatives, as well as any other wireless and wired protocols designated 3G, 4G, 5G, and above. Computing device 500 may include multiple communication packets 506. For example, the first communication packet 506 may be dedicated to shorter-range wireless communications, such as Wi-Fi and Bluetooth, and the second communication packet 506 may be dedicated to longer-range wireless communications, such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, and others.

[0091] Camera 532, including any depth sensor or proximity sensor, is coupled to an optional image processor 536 to perform conversion, analysis, noise reduction, comparison, depth or distance analysis, image understanding, and other processes as described herein. Processor 504 is coupled to the image processor to utilize interrupt-driven processes, set parameters, and control the operation of the image processor and camera. Image processing may alternatively be performed in processor 504, graphics CPU 512, camera 532, or any other device.

[0092] In various implementations, computing device 500 can be a laptop computer, netbook, notebook computer, ultrabook, smartphone, tablet device, personal digital assistant (PDA), ultra-mobile PC, mobile phone, desktop computer, server, set-top box, entertainment control unit, digital camera, portable music player, or digital video recorder. The computing device can be fixed, portable, or wearable. In another implementation, computing device 500 can be any other electronic device that processes or records data for processing elsewhere.

[0093] Implementations may utilize one or more memory chips, controllers, CPUs (central processing units), microchips, or integrated circuits interconnected via a motherboard, application-specific integrated circuit (ASIC), and / or field-programmable gate array (FPGA). The term "logic" may include, for example, software or hardware and / or a combination of software and hardware.

[0094] The references to "an embodiment," "an embodiment," "an example embodiment," "various embodiments," etc., indicate that the embodiments described in this way may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes that particular feature, structure, or characteristic. Furthermore, some embodiments may have some, all, or none of the features described with respect to other embodiments.

[0095] In the following description and claims, the term “coupling” and its derivatives may be used. “Coupling” is used to indicate that two or more elements cooperate or interact with each other, but there may or may not be an intervening physical or electrical component between them.

[0096] For the purposes of the claims, unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to describe common elements merely indicates that different instances of similar elements are being referred to, and is not intended to imply that the elements described in this way must be in a given sequence in time, space, rank, or any other way.

[0097] The accompanying drawings and the foregoing description provide examples of embodiments. Those skilled in the art will understand that one or more of the described elements can be combined into a single functional element. Alternatively, certain elements can be divided into multiple functional elements. Elements from one embodiment can be added to another embodiment. For example, the order of the processes described herein can be changed and is not limited to the manner described herein. Furthermore, the actions in any flowchart need not be performed in the order shown; nor is it necessary to perform all actions. Additionally, those actions that do not depend on other actions can be performed in parallel with other actions. The scope of the embodiments is by no means limited to these specific examples. Many variations, whether or not explicitly given in the specification, such as differences in structure, size, and material use, are possible. The scope of the embodiments is at least as broad as given by the appended claims.

[0098] Embodiments may be provided, for example, as computer program products that may include one or more transient or non-transitory machine-readable storage media having machine-executable instructions stored thereon, which, when executed by one or more machines such as a computer, computer network, or other electronic device, cause the one or more machines to perform operations according to the embodiments described herein. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disc read-only memory), magneto-optical disks, ROMs, RAMs, EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic cards or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0099] Figure 6A A transaction sequence 600 on a panoramic video system with a hemispherical cube map is illustrated according to one embodiment. For the sake of brevity, previous references will not be discussed or repeated below. Figures 1-5 Numerous details are mentioned or discussed. Furthermore, it is envisioned that any processing or transaction utilizing this and other diagrams can be executed by processing logic, which may include hardware (e.g., circuits, special-purpose logic, programmable logic, etc.), software (e.g., instructions running on a processing device), or a combination thereof, as determined by… Figure 1 One or more of the forward projection mechanism 110 and / or forward projection components 120, 130 and / or Figure 2 The reverse projection mechanism 260 and / or one or more of the reverse projection components (not shown) are used to facilitate this. For the sake of simplicity and clarity, any processes or transactions associated with the diagram and other diagrams may be illustrated or described in a linear sequence; however, it is contemplated that any of them may be executed in parallel, asynchronously, or in a different order.

[0100] like Figure 6A As shown, at server computing device 100, camera 242 is used to capture images and / or videos, as previously referenced. Figure 2 The captured information is then provided for use by Figure 1 Forward projection mechanism 110 facilitates forward projection (projection mapping, etc.) 601. Once this forward projection mapping is performed, the resulting data is then encoded using video encoder 213 and transmitted to video decoder 273 at client computing device 250. The encoded data is then decoded using video decoder 273 and forwarded for reverse projection (viewport generation, format creation, etc.) 611, facilitated by reverse projection mechanism 260, as referenced. Figure 2 Further description.

[0101] For example, as determined and selected by position selector 613 and by Figure 2 The location selection and display logic 268 facilitates, in one embodiment, the creation of a hemispherical content format (e.g., a 180°×180° hemispherical format in fisheye or ERP format content), or in another embodiment, the generation of a viewport (less than 180°×180°) for a user to view via display device 277.

[0102] Figure 6B A method 620 for forward projection is illustrated according to one embodiment. For the sake of brevity, previous references will not be discussed or repeated below. Figures 1-6A Numerous details are mentioned or discussed. Furthermore, it is envisioned that any processing or transaction utilizing this and other diagrams can be executed by processing logic, which may include hardware (e.g., circuits, special-purpose logic, programmable logic, etc.), software (e.g., instructions running on a processing device), or a combination thereof, as determined by… Figure 1 The forward projection mechanism 110 and / or one or more of the forward projection components 120, 130 are used to facilitate this. For the sake of simplicity and clarity, any processes or transactions associated with the diagram and other diagrams may be illustrated or recorded in a linear sequence; however, it is contemplated that any of them may be executed in parallel, asynchronously, or in a different order.

[0103] Method 620 begins at block 621, where 180°×180° content in a format such as fisheye, ERP, etc., is initiated at the server computing device. At block 623, this format is then mapped to a layout that provides or has a full face and multiple half-faces, for example, four half-faces representing a right half, left half, top half, and bottom half. In one embodiment, at block 625, this mapping is then processed to arrange the full face and four half-faces in a single row, including rotations of two half-faces. At block 627, this arrangement is enhanced using active padding (e.g., horizontal padding, vertical padding, etc.) from the sphere's neighbors. At block 629, this arrangement is then encoded at the server computing device using a video encoder, and then at block 631, the encoded arrangement is transmitted to the client computing device using one or more networks (e.g., a neighborhood network, a cloud network, the Internet, etc.).

[0104] Figure 6C A method 650 for reverse projection is illustrated according to one embodiment. For the sake of brevity, previous references will not be discussed or repeated below. Figures 1-6B Numerous details are mentioned or discussed. Furthermore, it is envisioned that any processing or transaction utilizing this and other diagrams can be executed by processing logic, which may include hardware (e.g., circuits, special-purpose logic, programmable logic, etc.), software (e.g., instructions running on a processing device), or a combination thereof, as determined by… Figure 2The reverse projection mechanism 260 and / or one or more of the reverse projection components (not shown) are used to facilitate this. For the sake of simplicity and clarity, any processes or transactions associated with the diagram and other diagrams may be illustrated or described in a linear sequence; however, it is contemplated that any of them may be executed in parallel, asynchronously, or in a different order.

[0105] Method 650 begins at block 651, wherein the client computing device receives data from the server computing device. Figure 6B The encoded layout is then decoded by the video decoder in block 653, and once decoded, the filled area of ​​the layout is clipped or blended in block 655. In block 657, the full front and four hemispheres of the texture are projected onto the hemisphere. In block 659, it is determined whether to create a hemispherical format in fisheye or ERP format content or to generate a viewport for viewing on a display device. If a hemispherical format is selected, the hemispherical format is generated in block 667. If a viewport is selected, the viewport is generated in block 669.

[0106] The following terms and / or examples are further embodiments or examples. Specific details in the examples can be used anywhere in one or more embodiments. Various features of different embodiments or examples can be combined in a variety of ways, including some features and excluding others, to suit a variety of different applications. Examples may include, for example: methods for facilitating hybrid communication according to the embodiments and examples described herein, means for performing actions of the method, at least one machine-readable medium including instructions that, when executed by a machine, cause the machine to perform the actions of the method, or means or systems for facilitating hybrid communication.

[0107] Some embodiments relate to Example 1, which includes an apparatus for facilitating an imaging environment in a hemispherical cubemap projection format. The apparatus includes one or more processors coupled to a memory, the processors configured to: capture an image with image content via a camera, wherein the image content represented by the image is omnidirectional, such that the image content, although represented smaller than a sphere, is mapped onto the sphere; map the image to a cube representation based on the six faces of the cube, wherein one or more of the six faces are classified as inactive regions, such that the one or more faces remain unoccupied or partially occupied; and arrange the six faces in a compact representation based on the cube representation by avoiding including the inactive regions.

[0108] Example 2 includes a subject as described in Example 1, wherein the six faces are arranged in a single row and include one full face and four partial faces.

[0109] Example 3 includes the subject matter as described in Examples 1-2, wherein the one or more processors are further configured to apply fill to one or more regions having effective neighboring regions represented on the sphere, and to avoid filling other regions that lack one or more of the effective neighboring regions and provide partial sphere representation.

[0110] Example 4 includes themes as described in Examples 1-3, wherein the cube representation is based on one or more of the Hemisphere Cube Mapping Format (HCMF) and Hemisphere Isometric Cube Mapping Format (HEAC), wherein the hemisphere is smaller than the sphere and is represented by a full face and four half faces, wherein the fill is further applied to the hemisphere.

[0111] Example 5 includes the subject matter as described in Examples 1-4, wherein the one or more processors further utilize a video encoder to encode the arrangement and transmit the encoded arrangement to a computing device via one or more networks, wherein the one or more processors, which are co-located on a common semiconductor package, include one or more of a central processing unit and a graphics processing unit.

[0112] Some embodiments relate to Example 6, which includes an apparatus for facilitating an imaging environment in a hemispherical cubemap projection format. The apparatus includes one or more processors coupled to a memory, the processors being configured to: receive an encoded arrangement of six faces of a cube representation of an image having image content, wherein one or more of the six faces are classified as inactive areas such that the one or more faces remain unoccupied or partially occupied; and decode the encoded arrangement into a decoded arrangement via a video decoder.

[0113] Example 7 includes the subject matter as described in Example 6, wherein the arrangement provides a compact representation of the cube representation by avoiding the inclusion of the inactive regions when encoded or decoded.

[0114] Example 8 includes the subject as described in Examples 6-7, wherein the cube representation is based on one or more of the Hemispherical Cube Mapping Format (HCMF) and the Hemispherical Isometric Cube Mapping Format (HEAC), wherein a hemisphere is smaller than a full sphere and is represented by a full face and four half faces.

[0115] Example 9 includes the subject matter as described in Examples 6-8, wherein the one or more processors, which are co-located on a common semiconductor package, include one or more of a central processing unit and a graphics processing unit.

[0116] Some embodiments relate to Example 10, which includes a method for facilitating an imaging environment using a hemispherical cubemap projection format. The method includes: capturing an image with image content by a camera coupled to one or more processors, wherein the image content represented by the image is omnidirectional, such that the image content, although represented smaller than a sphere, is mapped onto the sphere; mapping the image to a cube representation based on the six faces of the cube, wherein one or more of the six faces are classified as inactive regions, such that the one or more faces remain unoccupied or partially occupied; and arranging the six faces in a compact representation based on the cube representation by avoiding including the inactive regions.

[0117] Example 11 includes a subject as described in Example 10, wherein the six faces are arranged in a single row and include one full face and four partial faces.

[0118] Example 12 includes the subject as described in Examples 10-11, and also includes applying fill to one or more regions having effective neighboring regions represented on the sphere, and avoiding filling other regions that lack one or more of the effective neighboring regions and provide partial sphere representation.

[0119] Example 13 includes themes as described in Examples 11-12, wherein the cube representation is based on one or more of the Hemisphere Cube Mapping Format (HCMF) and Hemisphere Isometric Cube Mapping Format (HEAC), wherein the hemisphere is smaller than the sphere and is represented by a full face and four half faces, wherein the fill is further applied to the hemisphere.

[0120] Example 14 includes the subject matter as described in Examples 11-13, and further includes encoding the arrangement using a video encoder and transmitting the encoded arrangement to a computing device via one or more networks, wherein the one or more processors, which are co-located on a common semiconductor package, include one or more of a central processing unit and a graphics processing unit.

[0121] Some embodiments relate to Example 15, which includes a data processing system comprising a memory; and one or more processors coupled to the memory, the processors being configured to: capture an image having image content by a camera, wherein the image content represented by the image is omnidirectional, such that the image content, although represented smaller than a sphere, is mapped onto the sphere; map the image to a cube representation based on the six faces of a cube, wherein one or more of the six faces are classified as inactive regions, such that the one or more faces remain unoccupied or partially occupied; and arrange the six faces in a compact representation based on the cube representation by avoiding including the inactive regions.

[0122] Example 16 includes a subject as described in Example 15, wherein the six faces are arranged in a single row and include one full face and four partial faces.

[0123] Example 17 includes the subject matter as described in Examples 15-16, wherein the one or more processors are configured to apply fill to one or more regions having effective neighboring regions represented on the sphere, and to avoid filling other regions that lack one or more of the effective neighboring regions and provide partial sphere representation.

[0124] Example 18 includes themes as described in Examples 15-17, wherein the cube representation is based on one or more of the Hemisphere Cube Mapping Format (HCMF) and Hemisphere Isometric Cube Mapping Format (HEAC), wherein the hemisphere is smaller than the sphere and is represented by a full face and four half faces, wherein the fill is further applied to the hemisphere.

[0125] Example 19 includes the subject matter as described in Examples 15-18, wherein one or more processors encode the arrangement using a video encoder and transmit the encoded arrangement to a computing device via one or more networks, wherein the one or more processors, which are co-located on a common semiconductor package, include one or more of a central processing unit and a graphics processing unit.

[0126] Some embodiments relate to Example 20, which includes an apparatus for facilitating an imaging environment in a hemispherical cubemap projection format. The apparatus includes: means for capturing an image having image content via a camera coupled to one or more processors, wherein the image content represented by the image is omnidirectional, such that the image content is represented smaller than a sphere but mapped onto the sphere; means for mapping the image to a cube representation based on the six faces of the cube, wherein one or more of the six faces are classified as inactive regions, such that the one or more faces remain unoccupied or partially occupied; and means for arranging the six faces in a compact representation based on the cube representation by avoiding including the inactive regions.

[0127] Example 21 includes a subject as described in Example 20, wherein the six faces are arranged in a single row and include one full face and four partial faces.

[0128] Example 22 includes the subject matter as described in Examples 20-21, and further includes means for applying fill to one or more regions having effective neighboring regions represented on the sphere and avoiding filling other regions that lack one or more of the effective neighboring regions and provide partial representation of the sphere.

[0129] Example 23 includes themes as described in Examples 20-22, wherein the cube representation is based on one or more of the Hemisphere Cube Mapping Format (HCMF) and Hemisphere Isometric Cube Mapping Format (HEAC), wherein the hemisphere is smaller than the sphere and is represented by a full face and four half faces, wherein the fill is further applied to the hemisphere.

[0130] Example 24 includes the subject matter as described in Examples 20-23, and further includes means for encoding the arrangement using a video encoder and transmitting the encoded arrangement to a computing device via one or more networks, wherein the one or more processors, which are co-located on a common semiconductor package, include one or more of a central processing unit and a graphics processing unit.

[0131] Example 25 includes at least one non-transitory or tangible machine-readable medium comprising a plurality of instructions which, when executed on a computing device, implement or perform the method claimed in any of the claims or Examples 6-14.

[0132] Example 26 includes at least one machine-readable medium comprising a plurality of instructions which, when executed on a computing device, implement or perform the method claimed in any of the claims or Examples 6-14.

[0133] Example 27 includes a system comprising an apparatus for implementing or performing the method claimed in any of the claims or Examples 6-14.

[0134] Example 28 includes an apparatus comprising means for performing the method claimed in any of the claims or Examples 6-14.

[0135] Example 29 includes a computing device arranged to implement or perform the method claimed in any of the claims or Examples 6-14.

[0136] Example 30 includes a communication device arranged to implement or perform the method claimed in any of the claims or Examples 6-14.

[0137] Example 31 includes at least one machine-readable medium comprising a plurality of instructions which, when executed on a computing device, implement or perform a method as claimed in any of the preceding claims or implement an apparatus as claimed in any of the preceding claims.

[0138] Example 32 includes at least one non-transitory or tangible machine-readable medium comprising a plurality of instructions which, when executed on a computing device, implement or perform a method as claimed in any of the preceding claims or implement an apparatus as claimed in any of the preceding claims.

[0139] Example 33 includes a system comprising a mechanism for implementing or performing a method as claimed in any of the preceding claims or an apparatus for implementing a method as claimed in any of the preceding claims.

[0140] Example 34 includes an apparatus comprising means for performing a method as claimed in any of the preceding claims.

[0141] Example 35 includes a computing device arranged to implement or perform any method as claimed in the preceding claims or to implement any means as claimed in the preceding claims.

[0142] Example 36 includes a communication device arranged to implement or perform any method as claimed in the preceding claims or to implement any means as claimed in the preceding claims.

[0143] The accompanying drawings and the foregoing description provide examples of embodiments. Those skilled in the art will understand that one or more of the described elements may be combined into a single functional element. Alternatively, certain elements may be divided into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, the order of processes described herein may be changed and is not limited to the manner described herein. Furthermore, the actions in any flowchart need not be performed in the order shown; nor is it necessary to perform all actions. Moreover, those actions that do not depend on other actions may be performed in parallel with other actions. The scope of the embodiments is by no means limited to these specific examples. Many variations, whether or not explicitly given in the specification, such as differences in structure, size, and material use, are possible. The scope of the embodiments is at least as broad as given by the appended claims.

Claims

1. An encoder, comprising: The first circuit is used for: A hemispherical image is mapped to a hemispherical cube map, the hemispherical cube map including a complete face and a first half face, a second half face, a third half face and a fourth half face, the complete face and the first half face, the second half face, the third half face and the fourth half face representing the same hemisphere; The complete surface and the first, second, third, and fourth half-faces are arranged in a one-dimensional data arrangement, which sequentially includes the first half-face, the second half-face, the complete surface, the third half-face, and the fourth half-face, wherein at least two of the first, second, third, and fourth half-faces are rotated; and The encoding includes the message containing the hemispherical cube map; as well as The second circuit is used to output the message.

2. The encoder as claimed in claim 1, wherein, The first circuit is used to apply fill between the first half-face and the second half-face.

3. The encoder as described in claim 1 or 2, wherein, The hemispherical cube map is based on one or more of the hemispherical cube map format and the hemispherical isoangular cube map format.

4. The encoder as claimed in claim 1 or 2, wherein, The first circuit and the second circuit are located on a common semiconductor package and include one or more of a central processing unit and a graphics processing unit.

5. An encoding method, comprising: A hemispherical image is mapped to a hemispherical cube map, the hemispherical cube map including a complete face and a first half face, a second half face, a third half face and a fourth half face, the complete face and the first half face, the second half face, the third half face and the fourth half face representing the same hemisphere; By using processor execution instructions, the complete face and the first, second, third, and fourth half-faces are arranged into a one-dimensional data arrangement. This one-dimensional data arrangement sequentially includes the first half-face, the second half-face, the complete face, the third half-face, and the fourth half-face, wherein at least two of the first, second, third, and fourth half-faces are rotated; and The encoding includes the message containing the hemispherical cube texture.

6. The method of claim 5, further comprising: Apply filler between the first half and the second half.

7. The method of claim 5 or 6, wherein, The hemispherical cube map is based on one or more of the hemispherical cube map format and the hemispherical isoangular cube map format.

8. The method of claim 5 or 6, wherein, The message is encoded using a video encoder, wherein the video encoder and the processor reside on a common semiconductor package, which includes one or more of a central processing unit and a graphics processing unit.

9. A machine-readable medium storing instructions that, when executed by a machine, cause the machine to perform the method as described in any one of claims 5 to 8.

10. An encoding device, comprising: Memory; The instructions in the device; At least one processor is configured to execute the instructions such that the at least one processor performs the method as described in any one of claims 5 to 8.

11. An encoding device comprising means for performing the method as described in any one of claims 5 to 8.

Images