Remote real-time three-dimensional image transmission system and method based on dual-camera stereo imaging

Abstract

The invention belongs to the technical field of remote image transmission, stereoscopic vision imaging and head-mounted display, and particularly relates to a remote real-time three-dimensional image transmission system and method based on double-camera stereoscopic imaging. And after dynamic calibration and parallax coding, the three-dimensional images are transmitted to a head-mounted display helmet through an extended communication protocol (such as Skype API), so that high-precision and low-delay remote interaction of the three-dimensional images is realized. The system solves the problem that traditional 2D transmission lacks depth information, and is suitable for scenes such as immersive remote cooperation and virtual reality social contact.

Description

Technical Field

[0001] This invention belongs to the fields of remote image transmission, stereoscopic vision imaging and head-mounted display technology, and particularly relates to a remote real-time three-dimensional image transmission system and method based on dual-camera stereoscopic imaging. Background Technology

[0002] Existing remote image transmission technologies (such as Skype and video conferencing systems) are mostly monocular 2D video, lacking depth information and unable to provide an immersive 3D experience. Traditional stereoscopic imaging devices (such as 3D cameras), while capable of capturing stereoscopic images, suffer from the following problems: 1. The transmission protocol is not optimized for 3D data, resulting in high latency and high bandwidth consumption; 2. The receiving display device (such as a 3D TV) requires a fixed scene, making it impossible to achieve a portable immersive experience; 3. Dual-camera synchronous calibration relies on manual adjustment, which is not accurate enough.

[0003] Therefore, there is an urgent need for a high-efficiency system that combines dual-camera stereo imaging, remote real-time transmission, and head-mounted display. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a remote real-time 3D image transmission system and method based on dual-camera stereo imaging.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A remote real-time 3D image transmission system based on dual-camera stereo imaging includes a front-end acquisition terminal, a remote communication link, and a back-end display terminal.

[0006] Preferably, the front-end acquisition terminal includes a dual-camera module, an image preprocessing unit, and a transmission encoding module.

[0007] Preferably, the dual-camera module is equipped with a CMOS image sensor, and the two cameras of the dual-camera module are arranged horizontally with a spacing that simulates the interpupillary distance of the human eye. The CMOS image sensor is used to acquire images from the left and right perspectives.

[0008] Preferably, the image preprocessing unit performs noise reduction and color correction on the left and right images, and generates an initial stereo image pair by predicting the disparity map through a deep neural network (DNN).

[0009] Preferably, the transmission encoding module encodes the stereo image pairs into H.265 3D format, adds metadata such as timestamps and calibration parameters, and transmits them to a remote location via Skype API or a dedicated communication protocol.

[0010] As a preferred option, its remote communication link is based on Skype or other real-time communication platforms, extending the transmission channel to support stereoscopic image data and achieving end-to-end encryption and bandwidth adaptation.

[0011] Preferably, the terminal display includes a head-mounted display helmet, a decoding and rendering module, and a human-computer interaction module.

[0012] As a preferred option, its head-mounted display helmet has built-in independent left and right OLED screens and is equipped with an inertial measurement unit to track head posture in real time.

[0013] As a preferred option, its decoding and rendering module receives the stereoscopic image stream, synchronizes the left and right images according to the timestamp, combines the helmet posture data to change the viewing angle, and generates an immersive 3D picture through the binocular rendering engine.

[0014] As a preferred option, its human-computer interaction module supports touch or gesture input and transmits control signals back to the front-end acquisition terminal.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The present invention relates to a system and method for real-time transmission of three-dimensional images to a head-mounted display helmet via a remote communication device (such as a modified version of Skype), thereby achieving high-precision, low-latency remote interaction of three-dimensional images. It pioneered the combination of dual-camera dynamic calibration and 3D streaming transmission, solving the problem of low accuracy in manual calibration of traditional equipment; Expand existing communication platforms (such as Skype) to support 3D streaming, reducing technology deployment costs; The integration of head-mounted displays and posture tracking enables immersive remote interaction, applicable to fields such as healthcare, education, and social interaction. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the system framework of the present invention; Figure 2 Flowchart for dual-camera calibration; Figure 3 A flowchart for optimizing the encoding of stereo images. Detailed Implementation

[0018] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0019] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0020] Example 1, as Figures 1-3 As shown, this invention provides a remote real-time 3D image transmission system based on dual-camera stereo imaging, specifically including: 1. Front-end data collection terminal: - Dual camera module: Two cameras are arranged horizontally with a spacing that simulates the interpupillary distance of the human eye (55-75mm adjustable). Each camera integrates a CMOS image sensor to simultaneously acquire images from the left and right perspectives. - Image preprocessing unit: Denoises and corrects the colors of the left and right images, predicts the disparity map through a deep neural network (DNN), and generates an initial stereo image pair; - Transmission encoding module: Encodes stereo image pairs into H.265 3D format, adds metadata such as timestamps and calibration parameters, and transmits them to a remote location via Skype API or a dedicated communication protocol (such as WebRTC extension).

[0021] 2. Remote communication link: - Based on Skype or other real-time communication platforms, expand the transmission channel to support stereoscopic image data (such as adding RTP payload format to identify 3D streams), and achieve end-to-end encryption and bandwidth adaptation.

[0022] 3. Terminal display end: - Head-mounted display helmet: Built-in independent left and right OLED screens (resolution ≥1080p / eye), equipped with an inertial measurement unit (IMU) to track head posture in real time; - Decoding and rendering module: Receives stereoscopic image stream, synchronizes left and right images according to timestamps, combines helmet posture data to change the viewing angle, and generates immersive 3D images through a binocular rendering engine; - Human-computer interaction module: Supports touch or gesture input and sends control signals back to the front-end acquisition end (such as adjusting the camera spacing and focus).

[0023] Core Methods 1. Dual-camera dynamic calibration algorithm: - The acquisition end automatically calculates the camera's intrinsic parameters (focal length, distortion coefficient) and extrinsic parameters (relative position, rotation matrix) using structured light or a checkerboard calibration board, generates a calibration matrix, and embeds it into the transmitted data; - The receiver compensates for geometric distortion based on the calibration matrix to ensure that the parallax matching accuracy of the left and right images is ≤0.5 pixels.

[0024] 2. Optimized stereoscopic image compression and transmission: - It adopts disparity-based layered coding, with the main camera image encoded as the base layer and the secondary camera image encoded as the enhancement layer, reducing bandwidth requirements (saving 30% bitrate compared to traditional 3D coding). - Introducing a hybrid transmission of Transmission Control Protocol (TCP) and User Datagram Protocol (UDP), key frames (I-frames) are guaranteed to be reliable through TCP, while motion frames (P / B-frames) are reduced in latency through UDP.

[0025] 3. Real-time rendering technology for head-mounted displays: - Developed based on the Unity / UE engine, supporting stereo rendering with a refresh rate of ≥90Hz and latency of ≤20ms; - By combining helmet IMU data, the display view is adjusted in real time (such as the image view shifts synchronously when the user turns their head) to enhance immersion.

[0026] Implementation Scenario 1: Remote Medical Surgical Guidance - Front end: Dual cameras (65mm spacing) above the operating table capture stereoscopic images of the surgical area, and the preprocessing unit annotates tissue depth information in real time; - Transmission: 3D streams are transmitted via Skype for Business extended channel, received by doctors wearing HoloLens 2 helmets, and can be zoomed / rotated in stereoscopic images using gestures; - Advantages: Doctors gain a true 3D view of the surgical area, improving guidance accuracy by 40%.

[0027] Implementation Scenario 2: Virtual Reality Social Interaction - Front-end: Users take selfies using dual cameras, generating 3D portraits with depth information; - Transmission: Transmitted to social platforms via a dedicated WebRTC channel, the receiving Meta Quest headset displays a dynamic 3D avatar and supports head tracking interaction; - Advantages: Social interaction and immersion are improved by 60% compared to 2D videos.

[0028] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A remote real-time three-dimensional image transmission system based on dual-camera stereo imaging, characterized in that, It includes the front-end data acquisition terminal, the remote communication link, and the back-end display terminal.

2. The remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 1, characterized in that, The front-end acquisition unit includes a dual-camera module, an image preprocessing unit, and a transmission encoding module.

3. The remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 2, characterized in that, The dual-camera module is equipped with a CMOS image sensor. The two cameras of the dual-camera module are arranged horizontally, with the spacing simulating the interpupillary distance of the human eye. The CMOS image sensor is used to acquire images from the left and right perspectives.

4. A remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 3, characterized in that, The image preprocessing unit performs noise reduction and color correction on the left and right images, and generates an initial stereo image pair by predicting the disparity map through a deep neural network (DNN).

5. A remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 4, characterized in that, The transmission encoding module encodes the stereo image pairs into H.265 3D format, adds metadata such as timestamps and calibration parameters, and transmits them to a remote location via Skype API or a dedicated communication protocol.

6. A remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 1, characterized in that, Its remote communication link is based on Skype or other real-time communication platforms, extending the transmission channel to support stereoscopic image data, and achieving end-to-end encryption and bandwidth adaptation.

7. A remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 1, characterized in that, The terminal display includes a head-mounted display helmet, a decoding and rendering module, and a human-computer interaction module.

8. A remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 1, characterized in that, Its head-mounted display helmet has built-in independent left and right OLED screens and is equipped with an inertial measurement unit to track head posture in real time.

9. A remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 1, characterized in that, Its decoding and rendering module receives the stereoscopic image stream, synchronizes the left and right images according to the timestamp, combines the helmet posture data to change the perspective, and generates an immersive 3D picture through the binocular rendering engine.

10. A remote real-time three-dimensional image transmission system based on dual-camera stereo imaging according to claim 1, characterized in that, Its human-computer interaction module supports touch or gesture input and sends control signals back to the front-end acquisition terminal.

Images