End-to-end side-by-side image processing system and method

Abstract

The invention provides an end-to-end side-by-side image processing method, which adaptively designs exclusive information frames for various video streaming applications, and enables the exclusive information frames to be entrained in videos transmitted by the video streaming applications, so that a viewer device restores a 3D effect which is originally damaged by the video streaming applications on the premise of not modifying own behaviors of the video streaming applications.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to an end-to-end side-by-side image processing system and method. Background Art

[0002] Side-by-side imaging is a 3D imaging technology that consists of two images aligned side by side. These two images simulate the views seen by the viewer's left and right eyes respectively. They appear to be the same, but are actually slightly different. The difference between the two is integrated and processed by the brain, allowing the viewer to perceive the 3D effect.

[0003] As network technology advances and network speeds increase, video streaming applications, such as video conference software such as Zoom, Microsoft Teams, and Google Meet, and live streaming software such as Twitch, YouTube Live, and Open Broadcaster Software (OBS), are becoming more and more popular. When the video transmitter uses side-by-side images as input for such video streaming applications (for example, the user selects a 3D camera that outputs side-by-side images as the input source), the video streaming application will perform certain edits on the side-by-side images, including resizing and cropping the side-by-side images and adding user interface elements such as toolbars, function buttons, drop-down menus, and small windows, thereby destroying the 3D effect originally expected to be presented by the side-by-side images, and the user experience of the viewer is likely to be greatly reduced.

[0004] For computer device manufacturers, their customers hope to see autostereoscopic 3D content on their products. However, the above-mentioned video streaming application is developed by a third party, so it is impossible to solve the above-mentioned problem by modifying the behavior of the video streaming application itself.

[0005] Therefore, it is desired to have an end-to-end side-by-side image processing solution that can solve the problem of the 3D effect of the side-by-side image being destroyed by the video streaming application without modifying the behavior of the video streaming application itself. Summary of the invention

[0006] The embodiment of the present disclosure provides an end-to-end side-by-side image processing system, which includes a first computer device and a second computer device. The first computer device runs a video streaming application and a first program. The second computer device runs a video streaming application and a second program. The video streaming application instructs the first computer device to transmit a video to the second computer device via a network, and instructs the second computer device to generate an edited image sequence based on the video. The first program instructs the first computer device to obtain the side-by-side image sequence. The first program further instructs the first computer device to determine the position and range information of the non-side-by-side area in the edited image sequence according to the video streaming application, and to generate an information frame based on the position and range information and the first side-by-side image in the side-by-side image sequence. The generated information frame includes a header area and a corresponding designated image block. The header area contains the position and range information. The generated information frame is included in the video transmitted by the first computer device. The second program instructs the second computer device to parse the information frame to obtain the designated image block and the position and range information. The second program instructs the second computer device to replace the non-side-by-side area in each edited image in the edited image sequence with the designated image block to generate an output image sequence. The second computer device displays the output image sequence on a display.

[0007] In one embodiment, the first program further enables the first computer device to indicate the position and range information in the header area by arranging image blocks of multiple colors, each color representing a specific number.

[0008] In one embodiment, the first program further enables the first computer device to generate a designated image block based on the first side-by-side image according to a scaling ratio of the side-by-side image sequence applied by the video streaming application.

[0009] In one embodiment, the first computer device obtains a side-by-side image sequence from a 3D camera.

[0010] In one embodiment, the first computer device further runs a third program that enables the first computer device to generate a side-by-side image sequence based on the 2D image sequence.

[0011] The embodiment of the present disclosure further provides an end-to-end side-by-side image processing method. The method is implemented by a first computer device and a second computer device. The first computer device and the second computer device run a video streaming application. The video streaming application instructs the first computer device to transmit the video to the second computer device via a network and instructs the second computer device to generate an edited image sequence based on the video. The method includes the step of obtaining the side-by-side image sequence by the first computer device. The method includes the step of determining the position and range information of the non-side-by-side area in the edited image sequence by the first computer device according to the video streaming application, and generating an information frame based on the position and range information and the first side-by-side image in the side-by-side image sequence. The generated information frame includes a header area and a corresponding designated image block. The header area contains position and range information. The generated information frame is included in the video transmitted by the first computer device. The method further includes the step of parsing the information frame by the second computer device to obtain the designated image block and the position and range information. The method further includes the step of replacing the non-side-by-side area in each edited image in the edited image sequence with a designated image block by the second computer device to generate an output image sequence. The method further comprises the step of displaying the output image sequence on a display.

[0012] The end-to-end side-by-side image processing system and method provided by the present disclosure adaptively designs exclusive information frames for various video streaming applications, and allows the information frames to be entrained in the video transmitted by the video streaming application, so that the viewing device can restore the 3D effect originally destroyed by the video streaming application without modifying the behavior of the video streaming application itself. Therefore, the auto-stereoscopic display function of the computer device in various video streaming applications is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present disclosure will be better understood from the following illustrative embodiments with accompanying drawings. In addition, it should be understood that in the flowchart of the present disclosure, the execution order of each block may be changed, and / or some blocks may be changed, deleted or merged.

[0014] Figure 1 Side-by-side images showing an example.

[0015] Figure 2 An edited image of an example is presented.

[0016] Figure 3 An example information frame is shown according to an embodiment of the present disclosure.

[0017] Figure 4 A system block diagram of an end-to-end side-by-side image processing system is shown according to an embodiment of the present disclosure.

[0018] Figure 5 According to an embodiment of the present disclosure, a flow chart of an end-to-end side-by-side image processing method is shown.

[0019] The reference numerals are described as follows:

[0020] 100: Side-by-side images

[0021] 101: Left image

[0022] 102: Right image

[0023] 200: Edited Image

[0024] 201: Toolbar

[0025] 202: Function button

[0026] 203: Small window

[0027] 211-214: Non-side-by-side area

[0028] 215: Side-by-side area

[0029] 300: Information frame

[0030] 310: Information frame indicator

[0031] 301-304: Header area

[0032] 311-314: Specify image blocks

[0033] 400: System

[0034] 401: First Computer Device

[0035] 402: Second computer device

[0036] 411: First storage unit

[0037] 412: Second storage unit

[0038] 421: First processing unit

[0039] 422: Second processing unit

[0040] 430: Video streaming applications

[0041] 431: First Procedure

[0042] 432: Second Procedure

[0043] 440: Network

[0044] 445: Streaming Server

[0045] 500: Methods

[0046] S501-S505: Steps DETAILED DESCRIPTION

[0047] The following description lists various embodiments of the present invention, but is not intended to limit the content of the present invention. The actual scope of the invention is defined by the claims.

[0048] In the various embodiments listed below, the same reference numerals will be used to represent the same or similar elements or components.

[0049] In this specification and claims, serial numbers, such as "first", "second", etc., are only for convenience of description and have no sequential relationship with each other.

[0050] The following description of the embodiments of the apparatus or system is also applicable to the embodiments of the method, and vice versa.

[0051] Figure 1 An exemplary side-by-side image 100 is shown, wherein the left image 101 and the right image 102 simulate the views seen by the left eye and the right eye of the viewer, respectively. Figure 1 As shown, the side-by-side images 100 appear to be symmetrical, but in fact there is a slight difference between the left image 101 and the right image 102. Such differences are integrated and processed by the brain, allowing the viewer to perceive a 3D effect.

[0052] It should be noted that Figure 1 The dotted line in FIG. 1 is only used to distinguish the left image 101 from the right image 102 and to clarify the concept of the side-by-side image 100 . Actually, the dotted line does not exist in the side-by-side image 100 .

[0053] When the transmitter uses the side-by-side image 100 as a broadcast object of a video streaming application, the video streaming application will perform some editing on the side-by-side image 100, including scaling, cropping, and adding user interface elements such as toolbars, function buttons, drop-down menus, and small windows to the side-by-side image before outputting it to the device of the viewer. In this article, the image output by the video streaming application is called an "edited image".

[0054] Figure 2 An example edited image 200 is presented. Figure 2As shown, the edited image 200 no longer looks almost symmetrical like the side-by-side image 100, because the video streaming application has added a toolbar 201, a function button 202, and a small window 203 thereon. More specifically, the toolbar 201 spans the segment 211 on the left and the segment 212 on the right of the edited image, making the segments 211 and 212 obviously asymmetrical; the function button 202 is in the segment 213 on the left of the edited image, making the segment 213 and its mapping segment on the right (note: not necessarily the segment 214) obviously asymmetrical; the small window 203 is in the segment 214 on the right of the edited image, making the segment 214 and its mapping segment on the left (note: not necessarily the segment 213) obviously asymmetrical. As a result, the edited image 200 is side-by-side only in the segment 215, and not in the segments 211-214. Therefore, zone 215 is referred to herein as a "side-by-side zone" and zones 211-214 are referred to herein as "non-side-by-side zones."

[0055] It should be noted that Figure 2 The dotted lines in FIG. 2 are only used to distinguish the non-side-by-side areas 211-214 from the side-by-side area 215 and to clarify the concept of the edited image 200. In fact, the dotted lines do not exist in the edited image 200.

[0056] It should be understood that Figure 2 This is only an example of an edited image, but it is not limiting. Different video streaming applications may have different user interface designs, so the distribution of non-side-by-side areas in the edited image may also be different. However, generally speaking, video streaming applications will configure user interface components in a relatively peripheral position in the edited image, so the non-side-by-side area will usually also be in a relatively peripheral position in the edited image. The non-side-by-side area is usually defined by a rectangle, but this disclosure is not limited to this.

[0057] In addition, the present disclosure does not strictly limit the position and range of the non-side-by-side area, except that the non-side-by-side area may include user interface elements that make the edited image 200 obviously asymmetrical. For example, the upper edge of the non-side-by-side area 213 may be 1 row, 3 rows, 5 rows, 10 rows, or more rows of pixels within a reasonable range above the upper edge of the function button 202, and the present disclosure does not strictly limit this.

[0058] Due to the existence of the non-side-by-side areas 211-214, the edited image 200 no longer has the 3D effect brought by the original side-by-side image 100. However, for computer device manufacturers, the video streaming application is developed by a third party, so they cannot tamper with the behavior of the video streaming application itself, and can only restore the original expected 3D effect by operating the input and output of the video streaming application. A straightforward solution is to remove the non-side-by-side areas 211-214 of the edited image 200 and only leave the side-by-side area 215 for output, or to scale the side-by-side area 215 to a suitable display ratio before output. This approach can indeed output a side-by-side image with a 3D effect, but the field of view (FoV) is reduced compared to the original side-by-side image 100. For example, the background and objects (such as the background and objects covered by the toolbar 201) Figure 1 The clocks and landscapes drawn would not be fully presented.

[0059] The end-to-end side-by-side image processing method provided by the present disclosure is intended to allow the video transmitted by the video streaming application to carry the information required for restoring the 3D effect, including the position and range information of the side-by-side areas 211-214, and the designated image blocks used to replace the non-side-by-side areas 211-214. Since the above information needs to be processed and transmitted by the video streaming application, it must be contained in a data format compatible with the video streaming application, that is, a video frame. In order to distinguish the concepts, this article refers to the video frame that carries the designated image block and the position and range information as an "information frame", and the video frame that transmits the side-by-side image as an "image frame". However, it should be noted that the information frame is essentially a video frame, but in this article, the information frame is defined as an image used to transmit the information required to restore the 3D effect, rather than an image like an image frame that is presented to the viewer.

[0060] Figure 3 According to an embodiment of the present disclosure, an example information frame 300 is shown. Figure 3 As shown, the information frame 300 may include an information frame indicator 310 , header areas 301 - 304 , and designated image blocks 311 - 314 .

[0061] The information frame indicator 310 is used to indicate that the information frame 300 is an information frame. Since the information frame 300 is entrained in the video, that is, in a sequence of video frames, and most of the video frames are image frames, the existence of the information frame indicator 310 helps the viewing device to identify the information frame from the video frame sequence, so as to parse it and obtain the information contained therein.

[0062] Header areas 301-304 correspond to Figure 2The non-side-by-side areas 211-214 in the image 200 are respectively provided with the position and range information of the non-side-by-side areas 211-214. The position and range information can be expressed in various forms, and the present disclosure is not limited thereto. In one embodiment, the position and range information can be expressed by the relative position of the upper left vertex of the non-side-by-side area in the edited image 200, in combination with the relative position of the lower right vertex of the non-side-by-side area in the edited image 200. For example, taking the upper left vertex of the edited image 200 as a reference, the position and range information can be expressed by the coordinates of the upper left vertex of the non-side-by-side area (x1-th row pixels, y1-th column pixels) in combination with the coordinates of the lower right vertex (x2-th row pixels, y2-th row pixels). In another embodiment, the position and range information can be expressed by the relative position of any vertex or center point of the non-side-by-side area in the edited image 200, in combination with the length and width of the non-side-by-side area. For example, taking the upper left vertex of the edited image 200 as a reference, the position and range information can be represented by the coordinates of the upper left vertex of the non-side-by-side area (x1th row pixels, y1th column pixels), the coordinates of the lower right vertex (x2th row pixels, y2th row pixels), the coordinates of the upper right vertex, the coordinates of the lower left vertex and any one of the center point coordinates, combined with the length L pixels and width W pixels of the non-side-by-side area.

[0063] In addition, it should be understood that although the position and range of the non-side-by-side area depends on the user interface design of the video streaming application, the video streaming application usually does not actively provide this information to other programs. Fortunately, for video streaming applications, the position of user interface elements in the window is usually fixed. Therefore, developers can observe the user interface design of various video streaming applications and adaptively design exclusive information frames for various video streaming applications.

[0064] The designated image blocks 311-314 correspond to the header areas 301-304, respectively, and are used to replace the non-side-by-side areas 211-214. The designated image blocks 311-314 can be obtained by scaling the image blocks corresponding to the non-side-by-side areas 211-214 in the side-by-side image 100. After receiving and parsing the information frame 300, the viewing device can obtain the designated image blocks 311-314 and the position and range information of the non-side-by-side areas 211-214. Afterwards, the non-side-by-side areas 211-214 in the edited image 200 can be replaced by the designated image blocks 311-314, respectively, to generate an output image with a side-by-side 3D effect.

[0065] Figure 4 According to an embodiment of the present disclosure, a system block diagram of an end-to-end side-by-side image processing system 400 is shown. Figure 4As shown, the image processing system 400 includes a first computer device 401 and a second computer device 402, which represent the devices of the broadcaster and the viewer respectively. The first computer device 401 transmits the video to the second computer device 402 via the network 440, which may pass through the streaming server 445. The first computer device 401 includes a first storage unit 411 and a first processing unit 421. The second computer device 402 includes a second storage unit 412, a second processing unit 422, and a display 450.

[0066] The first computer device 401 and the second computer device 402 can each be an electronic device capable of performing computing tasks, such as a personal computer such as a desktop computer or a laptop computer, or a mobile device such as a tablet computer or a smart phone, but the present disclosure is not limited thereto.

[0067] The first storage unit 411 and the second storage unit 412 may each be a storage device including a non-volatile memory, such as a hard disk, a solid state drive (SSD), a flash memory, or a read-only memory, but the present disclosure is not limited thereto.

[0068] The first processing unit 421 and the second processing unit 422 may each include any one or more general-purpose or special-purpose processors and combinations thereof for executing instructions, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a neural network processor (NNP) or other various types of processors and combinations thereof, but the present disclosure is not limited to this.

[0069] According to an embodiment of the present disclosure, the first storage unit 411 stores a video streaming application 430 and a first program 431, and the second storage unit stores a video streaming application 430 and a second program 432. The first program 431 and the second program 432 include a plurality of instructions corresponding to the side-by-side image processing method of the present disclosure. When the first processing unit 421 reads and runs the first program 431 from the first storage unit 411, the steps of the side-by-side image processing method of the present disclosure implemented on the first computer device 401 (i.e., the broadcaster) are executed. Similarly, when the second processing unit 422 reads and runs the second program 432 from the second storage unit 412, the steps of the side-by-side image processing method of the present disclosure are executed on the second computer device 402 (i.e., the viewer) are executed.

[0070] The video streaming application 430 may be any application involving video streaming, such as video conferencing software such as Zoom, Microsoft Teams, and Google Meet, or live streaming software such as Twitch, YouTube Live, and Open Broadcaster Software (OBS), but the present disclosure is not limited thereto. The video streaming application 430 may perform certain edits on the side-by-side images, including scaling and cropping the side-by-side images and adding user interface elements such as toolbars, function buttons, drop-down menus, and small windows, but the present disclosure is not limited to the user interface design of the video streaming application 430 itself.

[0071] The streaming server 445 may be a dedicated server for the video streaming application 430, responsible for encoding and compressing the video from the first computer device 401, and transmitting and distributing it to the second computer device 402 or other recipients. Depending on the system architecture design of the video streaming application 430, the above-mentioned editing operation on the side-by-side images may be partially or completely performed on any one or more of the first computer device 401, the second computer device 402 and the streaming server 445, but the present disclosure is not limited thereto.

[0072] It should be understood that although Figure 4 The same symbol 430 is used to represent the video streaming applications running on the first computer device 401 and the second computer device 402, but this only means that they are compatible, and does not mean that they must be exactly the same software program. For example, the video streaming application 430 running on the first computer device 401 and the second computer device 402 can be the same product project of the same manufacturer, but the program version (e.g., new version / old version, host / client) may be different, or different functional modules may be enabled by logging in with different roles (e.g., broadcaster / viewer).

[0073] The display 450 is a display device that can present the 3D effect brought by the side-by-side images to the viewer without the viewer using a wearable device (such as 3D / VR glasses or a helmet). In one embodiment, the display 450 may include a display controller and a display panel (although not shown in the figure). The display controller is configured to switch the display mode of the display panel (for example, allowing the viewer to switch between 2D mode and 3D mode) according to a display mode control signal from the second processing unit 422. The display panel can be implemented using any existing stereoscopic imaging technology, such as a parallax barrier, lenticular lenses, directional backlight, integral imaging, etc., but the present disclosure is not limited to this.

[0074] Although in Figure 4 In the illustrated embodiment, the display 450 is included in the second computer device 402, but the present disclosure is not limited thereto. In other embodiments, the second computer device 402 can display the output image on an external display.

[0075] Figure 5 According to an embodiment of the present disclosure, a flow chart of an end-to-end side-by-side image processing method 500 is shown. Figure 5 As shown, the method 500 may include steps S501-S505, wherein steps S501-S502 are steps executed by the first program 431 instructing the first computer device 401 (more specifically, the first processing unit 421), and steps S503-S505 are steps executed by the second program 432 instructing the second computer device 402 (more specifically, the second processing unit 422).

[0076] In step S501, a sequence of side-by-side images is obtained. For example, Figure 1 The side-by-side image 100 is a side-by-side image of a video frame, and a series of side-by-side images after the side-by-side image 100. The side-by-side image sequence is also referred to as a "side-by-side image sequence" herein.

[0077] In one embodiment, Figure 4 The first computer device 401 in the embodiment further includes or is connected to a 3D camera (although not shown in the figure) to obtain a side-by-side image sequence from the 3D camera. The 3D camera may include a lens and a processor. The lens may be composed of one or more lenses (including concave lenses, convex lenses, or lenses of various other forms) to capture images of scenes and objects in the environment. The processor may run firmware to perform various optimizations on the images captured by the lens, including noise reduction, color correction, contrast adjustment, etc. (but the present disclosure is not limited to this), and generate side-by-side images based on the optimized images.

[0078] In one embodiment, Figure 4 The first storage unit in further stores a third program (although not shown in the figure). When the first processing unit 421 runs the third program, the third program enables the first processing unit 421 to generate a side-by-side image sequence based on the 2D image sequence. The third program can be a module or plug-in for 3D visual applications related to virtual reality (VR) or mixed reality (AR), such as games, video players, distance learning software, virtual laboratories, etc., or an independent program for generating 3D side-by-side images based on 2D images, but the present disclosure is not limited to this. The third program may involve existing 3D imaging technologies such as virtual lens systems, convolutional neural networks (CNN), depth information extraction, 3D projection, etc., but the present disclosure is not limited to this.

[0079] In step S502, the position and range information of the non-side-by-side area in the edited image sequence is determined according to the video streaming application, and an information frame is generated based on the first side-by-side image in the side-by-side image sequence. For example, the first processing unit 421 determines the position and range information of each of the non-side-by-side areas 211-214 in the edited image 200 according to the video streaming application 430, and generates the information frame 300 based on the side-by-side image 100 in the side-by-side image sequence. As described above, the generated information frame 300 includes header areas 301-304 and their respective corresponding designated image blocks 311-314, wherein the header areas 301-304 each contain the position and range information of their corresponding designated image blocks 311-314. In addition, the generated information frame 300 is included in the video transmitted from the first computer device 401 to the second computer device 402.

[0080] In one embodiment, step S502 generates a designated image block based on the first side-by-side image according to the scaling ratio of the side-by-side image sequence by the video streaming application. For example, assuming that the scaling ratios of the length and width of the side-by-side image sequence by the video streaming application 430 are (α, β), the second processing unit 422 scales the length and width of the image blocks corresponding to the non-side-by-side areas 211-214 in the side-by-side image 100 by the ratio of (α, β) to generate designated image blocks 311-314.

[0081] In step S503, the information frame is parsed to obtain the designated image blocks and the position and range information. For example, the second processing unit 422 parses the information frame 300 to obtain the designated image blocks 311-314 and their corresponding position and range information.

[0082] Generally speaking, the header area of ​​an image usually represents information used to identify image attributes in the form of text, numbers or a combination thereof, so the header area of ​​an information frame may also use such a form to represent position and range information. However, for some video streaming applications, their special encoding and compression processes may cause the customized text and digital information in the header area to be distorted when parsed in step S503 compared to when generated in step S502. Therefore, in one embodiment, step S502 may use a format in the header area that is most compatible with the video streaming application and least likely to cause information distortion, that is, an image, to represent the position and range information. More specifically, an arrangement of image blocks of multiple colors may be used to represent the position and range information, where each color represents a specific number. The number of color types depends on the positional system used. For example, binary data uses two colors (such as black and white) to represent 0 and 1 respectively. Octal data uses eight colors (such as red, orange, yellow, green, blue, indigo, purple, and white) to represent 0-7 respectively. Hexadecimal data uses sixteen colors to represent 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F respectively.

[0083] The following shows an example of using an arrangement of image blocks of eight colors (ie, octal system) in a header to represent position and range information.

[0084]

[0085] W R G V W B O V W R Y B O I G Y

[0086] Each column of Table 1 represents an image block in the header area, where the letters R, O, Y, G, B, I, V, and W represent red, orange, yellow, green, blue, indigo, purple, and white, respectively, and these colors are mapped to numbers 0 to 7. Therefore, the arrangement of the eight color image blocks shown in Table 1, "WRGVWBOVWRYBOIGY", represents a string of numbers "7036741670241532".

[0087] Similarly, Figure 3 The information frame indicator 310 in may also be represented in a manner similar to that in Table 1, but the present disclosure is not limited thereto.

[0088] Please refer to Figure 5. In step S504, the non-side-by-side area in each edited image in the edited image sequence is replaced with a designated image block to generate an output image sequence. For example, the second processing unit 422 replaces the non-side-by-side areas 211-214 in the edited image 200 with designated image blocks 311-314 respectively to generate an output image. The same operation is performed on the consecutive edited image frames after the edited image 200 to generate an output image sequence. The generated output image sequence has the 3D effect of the side-by-side image.

[0089] In step S505, the output image sequence is displayed on a display, for example Figure 4 Display 450

[0090] In one embodiment, the above steps S502-S504 may be executed once at a fixed time to update the information frame, and even to update the designated image block used to replace the non-side-by-side area. The frequency of the fixed execution may be once every 1 second, 2 seconds, 3 seconds, 5 seconds, or any suitable frequency within a reasonable range, but the present disclosure is not limited thereto. For example, assuming that the frame rate of the output image is 24fps (frames per second), if the above steps S502-S504 are executed once per second to update the information frame 300, the updated designated image blocks 311-314 may be used to replace the non-side-by-side areas 211-214 in the edited image 200 and the subsequent 23 consecutive frames of edited images.

[0091] The end-to-end side-by-side image processing system and method provided by the present disclosure adaptively designs exclusive information frames for various video streaming applications, and allows the information frames to be entrained in the video transmitted by the video streaming application, so that the viewing device can restore the 3D effect originally destroyed by the video streaming application without modifying the behavior of the video streaming application itself. Therefore, the auto-stereoscopic display function of the computer device in various video streaming applications is improved.

[0092] The above paragraphs are described using multiple aspects. Obviously, the teachings of this article can be implemented in multiple ways, and any specific architecture or function disclosed in the examples is only a representative case. According to the teachings of this article, it should be understood in the art that each aspect disclosed in this article can be implemented independently, or two or more aspects can be combined to implement.

[0093] Although the present disclosure has been described above with reference to the embodiments, it is not intended to limit the present disclosure. Anyone skilled in the art may make slight changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the invention shall be determined by the following claims.

Claims

1. An end-to-end side-by-side image processing system, comprising: a first computer device running a video streaming application and a first program; as well as a second computer device running the video streaming application and a second program; The video streaming application enables the first computer device to transmit a video to the second computer device via a network, and enables the second computer device to generate an edited image sequence based on the video; wherein the first program enables the first computer device to obtain a side-by-side image sequence; wherein the first program further enables the first computer device to determine the position and range information of a non-side-by-side area in the edited image sequence according to the video streaming application, and to generate an information frame based on the position and range information and a first side-by-side image in the side-by-side image sequence, wherein the generated information frame includes a header area and a corresponding designated image block, wherein the header area contains the position and range information, wherein the generated information frame is included in the video transmitted by the first computer device; The second program enables the second computer device to parse the information frame to obtain the designated image block and the position and range information; wherein the second program further instructs the second computer device to replace the non-side-by-side area in each edited image in the edited image sequence with the designated image block to generate an output image sequence; and The second computer device displays the output image sequence on a display.

2. The system of claim 1, wherein the first program further causes the first computer device to represent the position and range information in the header area by arranging image blocks of multiple colors, wherein each color represents a specific number. 3 . The system of claim 1 , wherein the first program further enables the first computer device to generate the designated image block based on the first side-by-side image according to a scaling ratio of the side-by-side image sequence applied by the video streaming application.

4. The system of any one of claims 1 to 3, wherein the first computer device obtains the side-by-side image sequence from a 3D camera. 5 . The system of claim 1 , wherein the first computer device further runs a third program, the third program enabling the first computer device to generate the side-by-side image sequence based on a 2D image sequence.

6. An end-to-end side-by-side image processing method, implemented by a first computer device and a second computer device, wherein the first computer device and the second computer device run a video streaming application, wherein the video streaming application enables the first computer device to transmit a video to the second computer device via a network and enables the second computer device to generate an edited image sequence based on the video, the method comprising the following steps: Obtaining a side-by-side image sequence from the first computer device; The first computer device determines, according to the video streaming application, position and range information of a non-side-by-side area in the edited image sequence, and generates an information frame based on the position and range information and a first side-by-side image in the side-by-side image sequence, wherein the generated information frame includes a header area and a corresponding designated image block, wherein the header area contains the position and range information, wherein the generated information frame is included in the video transmitted by the first computer device; The second computer device parses the information frame to obtain the designated image block and the position and range information; replacing the non-side-by-side area in each edited image in the edited image sequence with the designated image block by the second computer device to generate an output image sequence; and The output image sequence is displayed on a display.

7. The method of claim 6, wherein generating the information frame based on the first side-by-side image in the side-by-side image sequence comprises: The position and range information is represented by an arrangement of image blocks of multiple colors in the header area, wherein each color represents a specific number.

8. The method of claim 6, wherein generating the information frame based on the first side-by-side image in the side-by-side image sequence comprises: Based on the first side-by-side image, the designated image block is generated according to a scaling ratio of the side-by-side image sequence applied by the video stream.

9. The method according to any one of claims 6 to 8, wherein obtaining the side-by-side image sequence comprises: The side-by-side image sequence is obtained from a 3D camera.

10. The method according to any one of claims 6 to 8, further comprising: The side-by-side image sequence is generated based on a 2D image sequence.

Images