Image transmission method suitable for virtual reality, image processing device, and image generation system

Abstract

This disclosure provides an image transmission method suitable for virtual reality, which includes recombining acquired first and second images to generate a third image suitable for transmission over physical lines, thus avoiding distortion of high-resolution images due to compression. This disclosure also includes an image processing apparatus and an image generation system that can utilize this image transmission method.

Description

Technical Field

[0001] This disclosure relates to an image transmission method, image processing apparatus, and image generation system applicable to virtual reality. In particular, it relates to a method, apparatus, and system for image processing that addresses transmission format limitations before transmitting high-resolution images via physical lines, thereby preventing the high-resolution images required for virtual reality from being compressed and distorted due to the limitations of physical line transmission formats. Background Technology

[0002] The development of wireless mobile network technology has provided opportunities for greater transmission bandwidth and more diverse application services. Virtual Reality (VR) is an achievement that combines computer graphics, computer simulation, artificial intelligence, sensing, display, and network processing technologies. It utilizes a three-dimensional virtual world to provide users with a fully perceptible artificial environment, including auditory and tactile experiences, creating a sense of immersion. Users can directly observe, manipulate, and touch the surrounding environment and the internal changes of objects, and interact with them, making them feel as if they are actually there. Furthermore, by combining video streaming technology, which is driven by the development of real-time high-speed wireless mobile networks, with virtual reality technology, the three-dimensional virtual world can present a real-time, immersive experience, allowing users to enjoy a realistic and immersive experience without being physically present. Summary of the Invention

[0003] One embodiment of this disclosure relates to an image transmission method applicable to virtual reality, comprising the following steps: acquiring a first image and a second image, both having an aspect ratio of 2:1; dividing the second image from a short side to generate two first sub-images with the same aspect ratio; dividing the first sub-image from a long side of one of the first sub-images to generate three second sub-images with the same aspect ratio; generating a third image, wherein the third image comprises the first image, the undivided first sub-images, and the three second sub-images, and one short side of the third image is composed of a short side of the first image and a short side of the undivided first sub-images, and the other short side of the third image is composed of the three short sides of the three second sub-images, such that the aspect ratio of the third image is 16:9; and transmitting the third image via a physical line.

[0004] Another embodiment of this disclosure relates to an image processing apparatus suitable for virtual reality, comprising an image input mechanism, an image processing mechanism, and an image output mechanism. The image input mechanism receives a first original image and a second original image acquired from a first photographic device. The image processing mechanism is transmittedly connected to the image input mechanism and includes an image processing unit for adjusting the first original image and the second original image into a first image and a second image, and the image processing unit further combines the first image and the second image into a third image having an aspect ratio of 16:9. The size of the third image is the sum of the dimensions of the first image and the second image. The image output mechanism is transmittedly connected to the image processing mechanism and outputs the third image via a physical line.

[0005] Another embodiment of this disclosure relates to an image generation system suitable for virtual reality, comprising a first photographic device, an image processing device, and a streaming encoder. The image processing device is transmittedly connected to the first photographic device and is used to adjust a plurality of original images acquired by the first photographic device into a plurality of virtual reality images, and to combine these virtual reality images to generate a transmitted image having an aspect ratio of 16:9. The streaming encoder is transmittedly connected to the image processing device via a physical line and is used to convert the transmitted image into a streaming signal. Attached Figure Description

[0006] When read in conjunction with the accompanying drawings, the following detailed description is the best way to understand the nature of this disclosure. It should be noted that, in accordance with standard industry practice, the various structures are not drawn to scale. In fact, the dimensions of the various structures can be arbitrarily increased or decreased for clarity of illustration.

[0007] Figure 1 This is a schematic diagram of the architecture of an image generation system according to some embodiments of the present disclosure.

[0008] Figure 2 This is a schematic diagram of the architecture of an image processing apparatus according to some embodiments of the present disclosure.

[0009] Figure 3 This is a schematic diagram of a combination of a first image and a second image according to some embodiments of the present disclosure.

[0010] Figure 4A These are schematic diagrams showing the resolution of a first image and a second image according to some embodiments of this disclosure.

[0011] Figure 4B This is a schematic diagram of the resolution of a segmented second image according to some embodiments of the present disclosure.

[0012] Figure 4C This is a schematic diagram of the resolution of a third image according to some embodiments of the present disclosure. Detailed Implementation

[0013] The following disclosure provides numerous different embodiments or instances of various components for implementing the provided subject matter. Specific examples of elements and configurations are described below to simplify this disclosure. Of course, these are merely examples and are not intended to be limiting. For example, an embodiment in which a first component is formed above or on a second component may include embodiments in which the first and second components are formed in direct contact, and may also include embodiments in which additional components may be formed between the first and second components such that the first and second components are not in direct contact. Additionally, element symbols and / or letters may be repeated in various instances of this disclosure. This repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or configurations discussed.

[0014] Furthermore, for ease of description, spatial relative terms such as "below," "under," "down," "above," "on," and similar terms may be used herein to describe the relationship of one element or component to another element or component(s), as illustrated in the figures. Spatial relative terms are intended to cover different orientations of the device in use or operation other than those depicted in the figures. The device may be oriented in other ways (rotated 90 degrees or otherwise) and therefore the spatial relative descriptors used herein may be interpreted in the same way.

[0015] As used herein, terms such as "first," "second," and "third" describe various elements, components, areas, layers, and / or segments, and these elements, components, areas, layers, and / or segments should not be limited by these terms. These terms may be used only to distinguish one element, component, area, layer, or segment from another. Unless clearly indicated by the context, terms such as "first," "second," and "third" as used herein do not imply a sequence or order.

[0016] This disclosure, in some embodiments, provides an image processing method suitable for existing hardware environments for building virtual reality—especially virtual reality combined with real-time streaming technology—to ensure that the virtual reality images experienced by the user have high image quality (e.g., at least 6K or higher). This ensures that the acquired real-time images can be accurately presented at the user's end after being transmitted through different methods such as physical lines and network streaming without loss of image quality, allowing the user to obtain a high-quality immersive experience.

[0017] like Figure 1The illustrated image generation system architecture diagram, in some embodiments, may include at least one first photographic device 101 to acquire dynamic, real-time images as at least a portion of the specific image content reconstructed within a virtual space established by virtual reality technology on the user's end. In some embodiments, the first photographic device 101 is a camera with at least two lenses. For example, the first photographic device 101 may include horizontally arranged double fisheye lenses to simulate the human left and right eyes and the distance between them, thereby acquiring images simulating different perspectives of the human left and right eyes. After these images are merged and processed using image processing techniques, a stereoscopic image with a sense of depth can be generated.

[0018] In some embodiments, the first photographic device 101 can capture 180-degree video, and in conjunction with the aforementioned horizontally arranged double fisheye lens, it can generate 180-degree stereoscopic video by simulating the perspectives seen by the human left and right eyes, for the purpose of creating a virtual environment with stereoscopic images.

[0019] In some embodiments, the image generation system may further include at least one second camera device 102. Each second camera device 102 may have the same hardware specifications as the first camera device 101, thus enabling the second camera device 102 to also capture 180-degree video. Furthermore, in some embodiments, the first camera device 101 and the second camera device 102 can provide different scene angles. For example, when broadcasting cultural performances or sporting events, the first camera device 101 and the second camera device 102 are positioned at different locations within the event venue to provide different scene angles for users to observe and experience. In another embodiment, the combination of the first camera device 101 and the second camera device 102 can be used to provide 360-degree video, that is, combining the 180-degree videos captured by the first camera device 101 and the second camera device 102 respectively into a 360-degree video. With more second camera devices 102, combinations of more pairs of two second camera devices 102 can also be used to obtain 360-degree videos from different shooting positions.

[0020] To enhance the realism of virtual reality, the original images generated by the first camera device 101 and the second camera device 102 must possess a certain level of resolution. In some embodiments, the first camera device 101 and the second camera device 102 must be able to output image quality standards of at least 4K. In some embodiments, the high-quality images from the first camera device 101 and the second camera device 102 have a shooting frame rate of at least 50fps. In some embodiments, the first camera device 101 and the second camera device 102 have a chroma sampling specification of at least 4:2:2. In some embodiments, the first camera device 101 and the second camera device 102 can support a color depth specification of at least 10 bits. In some embodiments, to accommodate the performance limitations of software operation or other hardware system specifications, the first camera device 101 and the second camera device 102 may provide other supported color depth specifications during actual operation, such as providing images with a color depth of 8 bits.

[0021] To achieve the goal of processing high-definition images into images that conform to virtual reality specifications (or VR images), such as... Figure 1 As shown, in some embodiments, the first photographic device 101 and the second photographic device 102 are connected to an image processing device 200 to process the original images obtained by the photographic devices into virtual reality. In some embodiments, the image processing device 200 is set up at a real-world location, such as a live performance or sporting event. In other words, for virtual reality users (or viewers), the first camera device 101, the second camera device 102, and the image processing device 200 are all devices located "far away." Virtual reality users can obtain an immersive experience using their "near-end" virtual reality devices through high-speed, high-bandwidth, and low-latency communication technologies, thus realizing a virtual reality live broadcast mode. For virtual reality live broadcast service providers, the first camera device 101, the second camera device 102, and the image processing device 200 at the "near-end" of the real-world location complete the acquisition of high-definition images, the virtualization of high-definition images, and the customization of special effects in video rendering. Then, through real-time streaming communication technology, virtual reality users located "far away" from the service provider can experience the live images in real time without having to be physically present at the real-world location, and simultaneously enjoy value-added functions such as special effects, free viewpoint switching, and viewing event data.

[0022] Since the first camera device 101, the second camera device 102, and the image processing device 200 are all located near the virtual reality live streaming service provider, in some embodiments, the camera devices such as the first camera device 101 and the second camera device 102 transmit data to the image processing device 200 via physical lines. In some embodiments, these camera devices and the image processing device 200 are connected via a Serial Digital Interface (SDI). In some embodiments, the SDI may be in the form of a board with four input ports, thus it can connect one first camera device 101 and one second camera device 102. That is, the left-eye and right-eye view images (101L, 101R) of the first camera device 101 and the left-eye and right-eye view images (102L, 102R) of the second camera device 102 are input to the image processing device 200 through different input ports of the SDI. In other embodiments, taking the use of eight photographic devices (one first photographic device 101 and seven second photographic devices 102) as an example, the raw images acquired by the photographic devices are input to the image processing device 200 using four serial digital interfaces 150. In some embodiments, the images captured by the photographic devices such as the first photographic device 101 and the second photographic device 102 are of 4K quality and are output to the image processing device 200 in the form of an SDI signal with a resolution of 3840*2160 and an aspect ratio of 16:9. However, this disclosure does not actually limit the specific specifications of the serial digital interfaces 150; in practice, suitable hardware devices can be used depending on the development of communication transmission interfaces.

[0023] In some embodiments, the first and second photographic devices 101 and 102 and the image processing device 200 include the transmission of signals such as SDI via coaxial cables. In some embodiments, the photographic devices and the image processing device 200 include fiber optic cables and / or SDI-to-fiber optic converters, meaning that the photographic devices and the image processing device 200 can also transmit signals via fiber optic cables or a combination of fiber optic cables and coaxial cables. In other embodiments, the more economical High Definition Multimedia Interface (HDMI) or DisplayPort (DP) and corresponding HDMI and DP cables can also be used for image transmission.

[0024] In some embodiments, the image processing apparatus 200 has the function of processing the original image into a virtual reality image in real time. Since the first camera device 101 and the second camera device 102 do not need to be virtual reality camera devices (i.e., the first camera device 101 and the second camera device 102 are not virtual reality camera devices), this also means that these camera devices themselves do not need to have the function of directly outputting virtual reality images. Therefore, in this disclosure, firstly, the actual processing of the image into a virtual reality image is completed by the image processing apparatus 200, which avoids the high cost of deploying a large number of virtual reality camera devices on the real scene; secondly, virtual reality camera devices are limited by their hardware specifications and can usually only produce virtual reality images of rough quality, which does not meet the actual image quality requirements; furthermore, if a virtual reality camera device that outputs virtual reality images itself is used, it means that it is difficult to perform customized post-processing and editing on the virtual reality image. Therefore, this disclosure uses the first camera device 101 and the second camera device 102, which are not virtual reality camera devices, to obtain the original image, and then the image processing apparatus 200 uniformly completes the post-processing of all images and makes them virtual reality.

[0025] In some embodiments, the image processing device 200 is a switcher capable of performing virtual reality processing on images. In some embodiments, such as... Figure 2 As shown, the image processing apparatus 200 disclosed herein, applicable to virtual reality, may include an image input mechanism 201, an image processing mechanism 202, and an image output mechanism 203. In some embodiments, the image input mechanism 201 is used to receive a first original image and a second original image (e.g., a left-eye view image 101L and a right-eye view image 101R that have not yet been virtualized) acquired by the first photographic device 101; if the image processing apparatus 200 is connected to at least one second photographic device 102, the image processing apparatus 200 may also receive a third original image and a fourth original image (e.g., a left-eye view image 102L and a right-eye view image 102R that have not yet been virtualized) acquired by the second photographic device 102. Just as the first original image and the second original image originate from different perspectives simulating the left and right eyes, the third original image and the fourth original image also correspond to different perspectives based on the same principle.

[0026] Since there is a possibility of using multiple imaging devices, in some embodiments disclosed herein, the image input mechanism 201 includes an input management unit (not shown in the figures). Therefore, in addition to the image from the first imaging device 101, the input management unit can be used to switch between different image sources, such as receiving a third original image and a fourth original image obtained from the second imaging device 102.

[0027] In some embodiments, through a first image processing unit 204 of the image processing mechanism 202, the first original image and the second original image can be adjusted into a first image 301 and a second image 302 that meet the requirements of virtual reality (hereinafter referred to as...). Figure 3 In some embodiments, the first image processing unit 204 is a graphics processing unit (GPU). For example, the adjustments made to the original image by the first image processing unit 204 of the image processing mechanism 202 may include image stitching processing; as mentioned above, the first photographic device 101 can capture 180-degree images, therefore in some embodiments, the left-eye view image 101L and the right-eye view image 101R captured by the first photographic device 101 are 180-degree images from different perspectives; and the left-eye view image 102L and the right-eye view image 102R captured by the second photographic device 102 are also 180-degree images from different perspectives. In some embodiments, the left-eye view images 101L and 102L are stitched together from the specifications of 180-degree images to form a first image 301 with a 360-degree specification, and the right-eye view images 101R and 102R are stitched together from the specifications of 180-degree images to form a second image 302 with a 360-degree specification. The first image 301 and the second image 302 are a type of 360-degree image containing two 180-degree images (but not a continuous 360-degree panoramic image, but a 360-degree image containing two 180-degree image contents). After being transmitted to the virtual reality device, the first image 301 and the second image 302 can be projected onto the left and right eye display mechanisms of the virtual reality device, respectively, so that the user can experience a stereoscopic virtual reality image. It should be noted that the first image and the second image disclosed herein are not limited to corresponding to the left eye view and the right eye view, respectively. The foregoing description is only for illustrative purposes, and the correspondence between the two can also be reversed.

[0028] In some embodiments, the image processing unit 202 may include a renderer 206 for rendering special effects onto the first image 301 and the second image 302. These special effects can encompass various aspects. For example, to convey the immersive experience of a real-life event to virtual reality users, special effects designed to enhance visual effects (VFX) can be rendered onto the first image 301 and the second image 302. For instance, different and customized special effects can be provided for artistic performances, concerts, or sporting events. Furthermore, the rendered special effects may also include data analysis information provided by third parties, such as weather information, match scores in sporting events, match analysis, player data, and sports science information, rendered onto the first image 301 and the second image 302. This allows virtual reality users to not only experience stereoscopic virtual reality images but also to view more information that enhances their experience in real time.

[0029] In some embodiments, the image processing unit 202 may include an operating or accessible virtual reality media library 208 to acquire image materials and import images such as general live broadcast footage, advertising images, close-up captures, slow motion, and highlight replays. In some embodiments, the virtual reality media library 208 may also provide different customized styles such as subtitles, flashcards, or backgrounds, as well as animation effects, thereby easily overlaying one or more layers onto the first image 301 and the second image 302 by combining the functions of the first image processing unit 204 of the image processing unit 202.

[0030] In some embodiments, the first image processing unit 204 of the image processing mechanism 202 can perform image processing procedures such as color correction, brightness contrast adjustment, dimming, position adjustment, cropping, and scaling on the first image 301, the second image 302, or image content such as special effects and added materials to be rendered. In some embodiments, a virtual scene can also be captured using a single camera device, and then real-time background removal can be performed through the first image processing unit 204 or by further combining the functions of the renderer 206 to realize the green screen image compositing function in the virtual reality scene.

[0031] As mentioned earlier, in order to improve the realism of virtual reality, the original images generated by the first camera device 101 and the second camera device 102 must have a certain level of resolution, for example, at least 4K image quality, or at least a resolution of 3840*2160. The images processed by the first image processing unit 204 of the image processing mechanism 202, such as... Figure 3As shown, for example, a first image 301 and a second image 302 formed by stitching together a 360-degree image can each have a resolution of at least 5760*2880. The first image 301 and the second image 302 can be combined into a virtual reality image 30 with an aspect ratio of 1:1 for use by a virtual reality device (the first image 301 and the second image 302 are displayed synchronously and respectively on the left and right eye display mechanisms of the virtual reality device). In some embodiments, the virtual reality image 30 has a resolution of at least 5760*5760, that is, at least 6K image quality. If the resolution of the virtual reality image 30 is too low, for example, if the long side of the first image 301 and the second image 302 contained therein is less than 5760 pixels and the image quality is less than 6K, then it will appear blurry in the visual environment of the virtual reality user, which will severely limit the user's experience. However, under the specifications of existing standards, the physical lines of the downstream streaming encoder do not support the 1:1 aspect ratio resolution format. Therefore, this disclosure proposes an image transmission method suitable for virtual reality to complete the output of virtual reality image 30 without affecting image quality.

[0032] Specifically, in some embodiments, the image processing device 200 includes an image output mechanism 203, which is transmittedly connected to the image processing mechanism 202 to output the image processed by the image processing mechanism 202 via physical lines. However, since the image processing device 200 must transmit the virtual reality image 30 through physical lines before converting it into a streaming signal, these physical lines may include a Serial Digital Interface (SDI) 150 that supports up to 8K resolution and corresponding SDI cables; or in other embodiments, they may include a High Definition Multimedia Interface (HDMI) or DisplayPort (DP) that also supports up to 8K resolution and corresponding cables. Taking SDI, which is commonly used for high-quality images, as an example, although the physical lines including SDI 150 and corresponding SDI cables can support 8K resolution and meet the requirements for transmitting high-resolution virtual reality images 30, these physical lines only support the transmission of image formats with an aspect ratio of 16:9, and are not compatible with virtual reality images 30 with an aspect ratio of 1:1. In other words, in practice, the resolution of high-quality images may be limited by the standards supported by the transmission interface used. Therefore, how to transmit high-quality images using transmission interfaces with transmission specification limitations, such as SDI, while also meeting the needs of transmitting high-quality images beyond the existing transmission specifications is one of the problems to be solved in the relevant technical field. In a comparative example, given that the SDI 150 only supports transmitting image formats with an aspect ratio of 16:9, the image processing device adjusts the 1:1 aspect ratio virtual reality image 30 to a 16:9 resolution format using lossy compression. For example, it compresses the virtual reality image 30 from 5760*5760 to 5760*3240. Then, the user's virtual reality device restores the compressed virtual reality image 30 from 5760*3240 back to the 1:1 aspect ratio of 5760*5760. However, in this comparative example, because some image information is lost during the compression of the virtual reality image 30 from 1:1 to 16:9, according to information theory... Information theory (also known as lossy compression) states that lost information cannot be recovered after lossy compression. Therefore, the image quality of the virtual reality image 30 restored by the user's virtual reality device will inevitably be worse than the original virtual reality image 30. The embodiments disclosed herein can solve the above problem by using an image processing device 200 to segment and recombine the virtual reality image 30 beforehand, while remaining compatible with existing physical lines supporting high-resolution image transmission (e.g., including a serial digital interface 150). The details will be described in detail below.

[0033] refer to Figure 4AThe first image 301 and the second image 302 generated by the first image processing unit 204 both have an aspect ratio of 2:1. They would originally form a virtual reality image 30 with a resolution format of 1:1 aspect ratio. However, in some embodiments disclosed herein, such as... Figure 4B As shown, the second image 302 is divided from its shorter side to generate two first sub-images 302a with the same aspect ratio, and one of the first sub-images 302a is divided from its longer side to generate three second sub-images 302b with the same aspect ratio. Thus, as... Figure 4C As shown, a third image 303 can be formed, which includes a first image 301, an undivided first sub-image 302a and three second sub-images 302b. One short side of the third image 303 is composed of one short side of the first image 301 and one short side of the undivided first sub-image 302a, and the other short side of the third image 303 is composed of the three short sides of the three second sub-images 302b, so that the aspect ratio of the third image is 16:9.

[0034] Taking a specific resolution as an example, in the above embodiment, the first image 301 and the second image 302 can have a resolution of 5760*2880. They would originally form a virtual reality image 30 with a resolution of 5760*5760, but... Figure 4C As shown, after the segmentation and recombination process described in the above embodiment, the long side of the first sub-image 302a with a resolution of 5760*1440 is aligned with the long side of the first image 301 with a resolution of 5760*2880; while the short sides of the three second sub-images 302b with a resolution of 1920*1440 are connected to the short sides of the first image 301 and the first sub-image 302a. After such recombination, the resulting third image 303 has a resolution of 7680*4320, which is an 8K image with an aspect ratio of 16:9, and can be transmitted through physical lines.

[0035] Since the third image 303 generated by the image processing apparatus 200 disclosed herein changes the aspect ratio of the image by segmentation and recombination, that is, only rearranging and combining the pixels of the image without any compression or restoration of the image quality, the size of the third image 303 used for transmission through physical lines is the sum of the size of the first image 301 and the second image 302.

[0036] In some embodiments, the step of segmenting the second image 302 and combining it with the first image 301 to form the third image 303 can be accomplished by a shader 210 running in the image processing unit 202. The shader 210 can generate instructions for the first image processing unit 204 to process and modify the pixels of the first image 301 and the second image 302. One advantage of using the shader 210 to generate the third image 303 is that it instructs the first image processing unit 204 in a low-level language, which can be highly efficient and avoids the need to use post-processing software to edit the image in a high-level language, thus reducing the computational power consumption of the first image processing unit 204.

[0037] In some embodiments, before the second image 302 is segmented, the aforementioned renderer 206 renders the effects onto the first image 301 and the second image 302. In other words, the generation of the third image 303, which is not a 1:1 aspect ratio image, is a special process performed specifically to meet the specification limitations of the corresponding physical circuitry. Therefore, before the third image 303 is generated, the first image 301 and the second image 302 should have completed all the necessary virtual reality processing and adjustments and additions to the visual effects.

[0038] In other embodiments, the segmented image may be a first image 301 instead of a second image 302. In other words, as long as either the first image 301 or the second image 302 is segmented, it can be reconstituted with the other unsegmented image into an image with an aspect ratio of 16:9 for transmission via physical lines. Furthermore, this disclosure... Figures 4A to 4C The disclosure is only for the purpose of demonstrating the segmentation and reconstruction results of the images, and does not limit the order of steps disclosed herein regarding the segmentation and reconstruction of virtual reality images by the shader 210 in response to the limitations of physical line transmission formats.

[0039] This disclosure, through the image segmentation and recombination technology disclosed above, cleverly arranges the virtual reality image 30 with an aspect ratio of 1:1 into an aspect ratio of 16:9. This not only does not degrade the image quality of the virtual reality image 30, but also makes full use of the bandwidth of the physical line, without generating any bandwidth waste during the transmission process.

[0040] In some embodiments, the image processing unit 202 includes a user interface (UI) that allows the operator of the image processing device to select the source of the image (e.g., a combination of the aforementioned first and second camera devices, or a combination of two other second camera devices), and to examine the image to confirm the rendered visual effects, the quality of the virtual reality, and to perform image selection and other related editing operations.

[0041] As mentioned above, the image output mechanism 203 is connected to the image processing mechanism 202 and can output the image processed by the image processing mechanism 202 via a physical line. This image is the third image 303. In some embodiments, the image processing device 200 may be a broadcasting machine with the function of virtualizing images. In some embodiments, the image processing device 200 may be a group of multiple interconnected computers, each used to run different functions to process the images waiting to be output.

[0042] For example Figure 1 As shown, in some embodiments, the image generation system includes a storage device 400, which is transmittedly connected to the image processing device 200 for storing the image output by the image processing device 200 (i.e., the third image 303). This storage device 400 is used for data backup, while in other embodiments, the image generation system may not include a storage device 400.

[0043] In some embodiments, the image generation system includes a streaming encoder 500, which is physically connected to the image processing device 200 and used to convert the third image 303 output by the image processing device 200 into a streaming signal. For example, the third image 303 with an aspect ratio of 16:9 can be encoded by the streaming encoder 500 into an HTTP Live Streaming (HLS) signal for further transmission to a remote location using existing wired or wireless network transmission architectures. For instance, 5G transmission technology suitable for high-resolution images can be used to transmit the HLS signal to a content delivery network (CDN) server 501 of a telecommunications service provider, allowing users to download these HLS signals instantly. The above process is merely an example of a signal transmission mode and method for transmitting images from a "far end" near the real scene, such as the first camera device 101, the second camera device 102, and the image processing device 200, to a "near end" near the virtual reality user in real time for the purpose of establishing a virtual space. However, this disclosure does not limit the above-mentioned technical solution for transmitting images through non-physical lines.

[0044] In some embodiments, the image generation system includes a virtual reality device 600, which receives streaming signals and decodes the streaming signals into images originally output from the image processing device 200. Due to limitations in signal format for physical circuits, the virtualized first image 301 and second image 302, after the aforementioned image segmentation and recombination, leave the image processing device 200 as a third image 303 with an aspect ratio of 16:9. This is then converted into a streaming signal by the streaming encoder 500 and finally transmitted to the virtual reality device 600. At this point, for the virtual reality user to correctly read the image, the streaming signal is not only decoded and reconverted into the third image 303, but also, because the third image 303 is a specific format for transmission, the decoded third image 303 needs to undergo another round of image segmentation and recombination to restore / regenerate the first image 301 and second image 302.

[0045] In some embodiments, the virtual reality device 600 includes a second image processing unit 601, which may be a graphics processing unit (GPU), and the virtual reality device 600 may run another shader (not shown) to generate instructions for the second image processing unit 601 to process and modify the pixels of the third image 303, making them as described above. Figure 4C The resolution diagram shown is restored to the previous one. Figure 3 The resolution diagram is shown. Since the second image processing unit 601 may be integrated into a consumer wearable device, the high efficiency and low power consumption of restoring the third image 303 to the first image 301 and the second image 302 through the shader are obvious.

[0046] In some embodiments, after the third image 303 is recombined into a virtual reality image 30 with an aspect ratio of 1:1, the first image 301 and the second image 302 in the virtual reality image 30 can be synchronously and respectively displayed on two display mechanisms of the virtual reality device 600, such as the left eye display mechanism 602L and the right eye display mechanism 602R, so that virtual reality users can feel a clear, three-dimensional and real-time on-site experience, and enjoy a high-specification sense of presence experience with the help of customized visual effects or data analysis information provided by third parties on the image.

[0047] In summary, in some embodiments, this disclosure provides an image generation system suitable for virtual reality to adjust a plurality of original images obtained by a photographic device into a plurality of virtual reality images via an image processing device suitable for virtual reality provided by this disclosure; and by using an image transmission method suitable for virtual reality provided by this disclosure, these virtual reality images are combined to generate a transmission image with an aspect ratio of 16:9, which can efficiently enable the transmission image to meet the format requirements of physical lines for image signals without sacrificing image resolution, thereby realizing the creation of high-resolution stereoscopic virtual reality real-time streaming services.

[0048] The foregoing outlines the structure of several embodiments, enabling those skilled in the art to better understand the nature of this disclosure. Those skilled in the art should understand that they can readily use this disclosure as a basis for designing or modifying other processes and structures to achieve the same purposes and / or advantages of the embodiments described herein. Those skilled in the art should also understand that such equivalent constructions do not depart from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications can be made herein without departing from the spirit and scope of this disclosure.

[0049] [Symbol Explanation]

[0050] 30: Virtual Reality Images

[0051] 101: The First Photographic Device

[0052] 101L: Left eye view image

[0053] 101R: Right Eye View Image

[0054] 102: Second Photographic Device

[0055] 102L: Left eye view image

[0056] 102R: Right eye view image

[0057] 150: Serial digital interface

[0058] 200: Image processing device

[0059] 201: Image Input Mechanism

[0060] 202: Image Processing Agency

[0061] 203: Image Output Mechanism

[0062] 204: First Image Processing Unit

[0063] 206: Renderer

[0064] 208: Virtual Reality Media Library

[0065] 210: Shaders

[0066] 301: First Image

[0067] 302: Second Image

[0068] 302a: First Sub-image

[0069] 302b: Second Sub-image

[0070] 303: Third Image

[0071] 400: Storage device

[0072] 500: Stream Encoder

[0073] 501: Server Error

[0074] 600: Virtual Reality Device

[0075] 601: Second Image Processing Unit

[0076] 602L: Left Eye Display Mechanism

[0077] 602R: Right Eye Display Mechanism

Claims

1. An image transmission method applicable to virtual reality, the image transmission method comprising: A first image and a second image are acquired, both having an aspect ratio of 2:

1. The second image is divided from one of its shorter sides to generate two first sub-images with the same aspect ratio. The first sub-image is divided by one of the long sides of the first sub-image to generate three second sub-images with the same aspect ratio; A third image is generated, wherein the third image includes the first image, an undivided first sub-image, and the three second sub-images, and one short side of the third image is composed of a short side of the first image and a short side of the undivided first sub-image, and the other short side of the third image is composed of the three short sides of the three second sub-images, such that the aspect ratio of the third image is 16:9; and The third image is transmitted via a physical line.

2. The image transmission method as claimed in claim 1, wherein one long side of the first image has at least 5760 pixels.

3. The image transmission method as claimed in claim 1, wherein the steps of segmenting the second image, segmenting the first sub-image, and generating the third image include an image processing device running a shader to reassemble the segmented second image and the first sub-image with the first image into the third image.

4. The image transmission method as described in claim 1, further comprising: The third image is transmitted via the physical line to a stream encoder, where the third image is converted into a stream signal by the stream encoder.

5. The image transmission method as described in claim 4, further comprising: Transmit the streaming signal to a virtual reality device; Decode the streaming signal into the third image; and The first image and the second image are generated using the third image.

6. The image transmission method of claim 5, wherein the step of generating the first image and the second image using the third image includes the virtual reality device running a shader to reassemble the third image to output the first image and the second image to two display mechanisms of the virtual reality device respectively.

7. The image transmission method of claim 1, further comprising, before the step of segmenting the second image: Render special effects onto the first image and the second image.

8. An image processing apparatus suitable for virtual reality, comprising: An image input mechanism for receiving a first original image and a second original image acquired from a first photographic device; An image processing mechanism, transmittedly connected to the image input mechanism, includes a first image processing unit for adjusting a first original image and a second original image into a first image and a second image, both having an aspect ratio of 2:

1. The first image processing unit further segments the second image from its short side to generate two first sub-images with the same aspect ratio, and segments the first sub-image from the long side of one of the first sub-images to generate three second sub-images with the same aspect ratio. This is used to combine the first image and the second image into a third image with an aspect ratio of 16:

9. The third image includes the first image, the undivided first sub-image, and the three second sub-images, wherein one short side of the third image is formed by one short side of the first image and one short side of the undivided first sub-image, and the other short side of the third image is formed by the three short sides of the three second sub-images; and An image output mechanism is transmittedly connected to the image processing mechanism and is used to output the third image via a physical line.

9. The image processing apparatus of claim 8, wherein the first original image and the second original image are stitched together and adjusted to become the first image and the second image respectively, and the first image and the second image are 360-degree virtual reality images from different perspectives.

10. The image processing apparatus of claim 8, wherein the image processing mechanism includes operating a shader for instructing the first image processing unit to segment the second image to combine it with the first image into the third image.

11. The image processing apparatus of claim 8, wherein the image processing apparatus includes running a renderer for rendering effects to the first image and the second image.

12. The image processing apparatus of claim 8, wherein the image input mechanism includes an input management unit for switching between receiving a third original image and a fourth original image acquired from a second photographic device, the third original image and the fourth original image corresponding to different viewpoints.

13. The image processing apparatus of claim 8, wherein the physical circuitry includes a serial digital interface, a high-definition multimedia interface, or a display interface.

14. An image generation system suitable for virtual reality, comprising: A first photographic device; An image processing device, transmittedly connected to a first photographic device, is used to adjust a plurality of original images acquired by the first photographic device into a plurality of virtual reality images. These virtual reality images have an aspect ratio of 1:

1. Each virtual reality image includes a first image and a second image, both with an aspect ratio of 2:

1. The image processing device further segments the second image from its short side to generate two first sub-images with the same aspect ratio, and segments the first sub-image from the long side of one of the first sub-images to generate three second sub-images with the same aspect ratio. The first and second images are then combined to generate a plurality of third images with an aspect ratio of 16:9 as a transmission image. The transmission image is output via a physical line connected to the image processing device. The third image includes the first image, the undivided first sub-image, and the three second sub-images, wherein one short side of the third image is formed by one short side of the first image and one short side of the undivided first sub-image, and the other short side of the third image is formed by the three short sides of the three second sub-images; and A streaming encoder, which is connected to the image processing device via the physical line, is used to convert the transmitted image into a streaming signal.

15. The image generation system of claim 14, further comprising a storage device, the storage device being transmissively connected to the image processing device for storing the transmissive image.

16. The image generation system of claim 14, further comprising a virtual reality device for receiving the streaming signal and decoding the streaming signal into the transmitted image.

17. The image generation system of claim 16, wherein the virtual reality device comprises: A second image processing unit is used to reassemble the transmitted images to output the virtual reality images; and Two display mechanisms are used to display these virtual reality images from different perspectives.

18. The image generation system of claim 14, wherein the physical circuitry includes a serial digital interface, a high-definition multimedia interface, or a display interface.

19. The image generation system of claim 14, wherein the first photographic device comprises two lenses for acquiring the original images from different perspectives.

20. The image generation system of claim 14, further comprising at least one second photographic device, the second photographic device being transmittedly connected to the image processing device.

Images