Video see-through method, device, terminal device and storage medium

Abstract

A video perspective method, device, equipment and medium are disclosed, the method comprises the following steps: acquiring first image data collected by a first camera and second image data collected by a second camera; performing stereo correction on the first image data to obtain third image data and a first projection matrix corresponding to the first camera; performing stereo correction on the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera; performing parallel correction on the third image data and the fourth image data based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix, to obtain first perspective image data and second perspective image data. The application realizes the correction of the images obtained by the binocular camera, aligns the left and right binocular pictures, and makes the picture parallel to the line of sight, thereby producing a stereoscopic effect and improving the user experience.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a video perspective method and device, a terminal device and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of science and technology, the virtual reality industry has ushered in explosive growth. A number of consumer-level mass production products have now reached nearly 10 million annual sales.

[0003] At present, common perspective implementation methods mainly include optical perspective and video perspective. The implementation method of video perspective is to capture real-time views of the real world through a camera, and then combine it with computer image technology together to present on an opaque display. In addition, the main purpose of the video perspective method is to correctly display the pictures captured by two cameras on the display screen, align the pixels of the pictures, produce a stereoscopic effect, and align with the real world, thereby simulating the real world. However, the video perspective method has the problems that the two cameras cannot be completely aligned during installation, resulting in picture alignment errors when displaying the binocular pictures, and not considering the parallel alignment effect of the pictures and the line of sight.

[0004] In summary, how to correct the images obtained by the binocular camera to align the left and right binocular pictures and make the pictures parallel to the line of sight, thereby producing a stereoscopic effect and improving user experience, has become a technical problem to be solved in the field of image processing technology. SUMMARY

[0005] The main purpose of the present application is to provide a video perspective method, device, terminal device and computer readable storage medium. The purpose is to correct the images obtained by the binocular camera to align the left and right binocular pictures and make the pictures parallel to the line of sight, thereby producing a stereoscopic effect and improving user experience.

[0006] In order to achieve the above purpose, the present application provides a video perspective method, which comprises:

[0007] obtaining first image data captured by a first camera and second image data captured by a second camera;

[0008] stereo correcting the first image data to obtain third image data and a first projection matrix corresponding to the first camera;

[0009] stereo correcting the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera;

[0010] parallel correction is performed on the third image data and the fourth image data based on the target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix, to obtain first perspective image data and second perspective image data.

[0011] Optionally, before the step of acquiring the first image data collected by the first camera and the second image data collected by the second camera, the method further comprises:

[0012] calibrating the first camera and the second camera to obtain a first intrinsic parameter of the first camera, a second intrinsic parameter of the second camera and a first extrinsic parameter between the first camera and the second camera;

[0013] based on the first intrinsic parameter, the second intrinsic parameter and the first extrinsic parameter, a first rotation matrix corresponding to the first camera and a second rotation matrix corresponding to the second camera are determined respectively.

[0014] Optionally, the step of performing stereo correction on the first image data to obtain third image data and a first projection matrix corresponding to the first camera comprises:

[0015] based on the first rotation matrix, performing stereo correction on the first image data to obtain third image data and a first projection matrix corresponding to the first camera;

[0016] the step of performing stereo correction on the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera comprises:

[0017] based on the second rotation matrix, performing stereo correction on the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera.

[0018] Optionally, the method further comprises:

[0019] calibrating the first camera and the inertial measurement unit to obtain a second extrinsic parameter between the first camera and the inertial measurement unit;

[0020] based on the second extrinsic parameter and the first rotation matrix, a target rotation matrix corresponding to the first camera and the second camera is determined.

[0021] Optionally, the step of determining the target rotation matrix corresponding to the first camera and the second camera based on the second extrinsic parameter and the first rotation matrix comprises:

[0022] determining a first coordinate system orientation of the inertial measurement unit according to the second extrinsic parameter;

[0023] determining a second coordinate system orientation of the first camera according to the first rotation matrix;

[0024] determining a target rotation matrix corresponding to the first camera and the second camera according to the first coordinate system orientation and the second coordinate system orientation.

[0025] Optionally, the step of performing parallel correction on the third image data and the fourth image data based on the target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix comprises:

[0026] determining a first mapping matrix corresponding to the first camera based on the target rotation matrix and the first projection matrix;

[0027] determining a second mapping matrix corresponding to the second camera based on the target rotation matrix and the second projection matrix;

[0028] performing parallel correction on the third image data based on the first mapping matrix;

[0029] performing parallel correction on the fourth image data based on the second mapping matrix.

[0030] Optionally, after the step of obtaining the first perspective image data and the second perspective image data, the method further comprises:

[0031] displaying the first perspective image data and the second perspective image data on a VR display screen, wherein the first perspective image data is the third image data after parallel correction, and the second perspective image data is the fourth image data after parallel correction.

[0032] In addition, to achieve the above object, the application further provides a video perspective device, which comprises:

[0033] an image acquisition module, configured to acquire first image data collected by a first camera and second image data collected by a second camera;

[0034] a first stereo correction module, configured to perform stereo correction on the first image data to obtain third image data and a first projection matrix corresponding to the first camera;

[0035] a second stereo correction module, configured to perform stereo correction on the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera;

[0036] The parallel correction module performs parallel correction on the third image data and the fourth image data based on the target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix, to obtain first perspective image data and second perspective image data.

[0037] In addition, to achieve the above object, the application further provides a terminal device, which comprises a memory, a processor and a video perspective program stored in the memory and executable on the processor, wherein the video perspective program of the terminal device, when executed by the processor, implements the steps of the video perspective method.

[0038] In addition, to achieve the above object, the application further provides a computer readable storage medium, which stores a video perspective program, wherein the video perspective program, when executed by a processor, implements the steps of the video perspective method.

[0039] The video perspective method, device, terminal device and computer readable storage medium provided by the embodiments of the application are as follows: the first image data collected by a first camera and the second image data collected by a second camera are acquired; the first image data is subjected to stereoscopic correction to obtain third image data and a first projection matrix corresponding to the first camera; the second image data is subjected to stereoscopic correction to obtain fourth image data and a second projection matrix corresponding to the second camera; the third image data and the fourth image data are subjected to parallel correction based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix, to obtain first perspective image data and second perspective image data.

[0040] The embodiments of the application acquire the first image data photographed by the first camera and the second image data photographed by the second camera, then perform stereoscopic correction on the first image data and the second image data to obtain a first projection matrix corresponding to the first camera and a second projection matrix corresponding to the second camera, and generate third image data and fourth image data, wherein the third image data is the first image data after stereoscopic correction, and the fourth image data is the second image data after stereoscopic correction, then perform parallel correction on the third image data and the fourth image data based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix, to obtain first perspective image data and second perspective image data. Thus, the application corrects the image acquired by the binocular camera, so that the left and right binocular pictures are aligned, and the pictures are parallel to the line of sight, thereby generating a stereoscopic effect and improving user experience. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1This is a schematic diagram of the device structure of the terminal device hardware operating environment involved in the embodiments of the present invention;

[0042] Figure 2 This is a flowchart illustrating the steps of the first embodiment of the video perspective method of the present invention;

[0043] Figure 3 This is a schematic diagram of the sensor layout involved in an embodiment of the video perspective method of the present invention;

[0044] Figure 4 This is a schematic diagram of image stereoscopic correction involved in an embodiment of the video perspective method of the present invention;

[0045] Figure 5 This is a schematic diagram of the camera coordinate system orientation involved in an embodiment of the video perspective method of the present invention;

[0046] Figure 6 This is a schematic diagram of the IMU coordinate system orientation according to an embodiment of the video perspective method of the present invention;

[0047] Figure 7 This is a schematic diagram showing the orientation of the rotated camera coordinate system according to an embodiment of the video perspective method of the present invention.

[0048] Figure 8 This is a schematic diagram of the image correction process involved in an embodiment of the video perspective method of the present invention;

[0049] Figure 9 This is a schematic diagram of the functional modules of an embodiment of the video perspective device of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiment of the present invention.

[0053] The terminal device in this embodiment of the invention can be a terminal device applied in the field of video perspective technology. Specifically, the terminal device can be a smartphone, PC (Personal Computer), tablet computer, portable computer, etc.

[0054] like Figure 1As shown, the terminal device can include a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection communication between the components. The user interface 1003 can include a display screen (DiSplay), an input unit such as a keyboard (Keyboard), and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a Wi-Fi interface). The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory) such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art can understand that Figure 1 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0056] As Figure 1 As shown, the memory 1005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a video perspective program.

[0057] In Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; and the processor 1001 can be used to call the video perspective program stored in the memory 1005 and perform the following operations:

[0058] Obtain first image data collected by a first camera and second image data collected by a second camera;

[0059] Stereo correct the first image data to obtain third image data and a first projection matrix corresponding to the first camera;

[0060] Stereo correct the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera;

[0061] Based on the target rotation matrix corresponding to the first camera and the second camera, the first projection matrix, and the second projection matrix, parallel correct the third image data and the fourth image data to obtain first perspective image data and second perspective image data.

[0062] Optionally, the processor 1001 can also be configured to invoke the video perspective program stored in the memory 1005, and perform the following operations before the step of acquiring the first image data corresponding to the first camera and the second image data corresponding to the second camera:

[0063] calibrating the first camera and the second camera to obtain a first intrinsic parameter of the first camera, a second intrinsic parameter of the second camera, and a first extrinsic parameter between the first camera and the second camera;

[0064] determining a first rotation matrix corresponding to the first camera and a second rotation matrix corresponding to the second camera based on the first intrinsic parameter, the second intrinsic parameter, and the first extrinsic parameter.

[0065] Optionally, the processor 1001 can also be configured to invoke the video perspective program stored in the memory 1005, and perform the following operations:

[0066] performing stereo rectification on the first image data based on the first rotation matrix to obtain third image data and a first projection matrix corresponding to the first camera;

[0067] performing stereo rectification on the second image data based on the second rotation matrix to obtain fourth image data and a second projection matrix corresponding to the second camera.

[0068] Optionally, the processor 1001 can also be configured to invoke the video perspective program stored in the memory 1005, and perform the following operations:

[0069] calibrating the first camera and the inertial measurement unit to obtain a second extrinsic parameter between the first camera and the inertial measurement unit;

[0070] determining a target rotation matrix corresponding to the first camera and the second camera based on the second extrinsic parameter and the first rotation matrix.

[0071] Optionally, the processor 1001 can also be configured to invoke the video perspective program stored in the memory 1005, and perform the following operations:

[0072] determining a first coordinate system orientation of the inertial measurement unit according to the second extrinsic parameter;

[0073] determining a second coordinate system orientation of the first camera according to the first rotation matrix;

[0074] determining a target rotation matrix corresponding to the first camera and the second camera according to the first coordinate system orientation and the second coordinate system orientation.

[0075] Optionally, the processor 1001 can also be configured to invoke the video see-through program stored in the memory 1005, and perform the following operations:

[0076] determine a first mapping matrix corresponding to the first camera based on the target rotation matrix and the first projection matrix;

[0077] determine a second mapping matrix corresponding to the second camera based on the target rotation matrix and the second projection matrix;

[0078] perform parallel correction on the third image data based on the first mapping matrix;

[0079] perform parallel correction on the fourth image data based on the second mapping matrix.

[0080] Optionally, the processor 1001 can also be configured to invoke the video see-through program stored in the memory 1005, and perform the following operations after the step of obtaining the first see-through image data and the second see-through image data:

[0081] display the first see-through image data and the second see-through image data on a VR display screen, wherein the first see-through image data is the third image data after parallel correction, and the second see-through image data is the fourth image data after parallel correction.

[0082] Based on the terminal device described above, embodiments of the video see-through method are proposed.

[0083] At present, common see-through implementation methods mainly include optical see-through and video see-through. The video see-through is implemented by capturing real-time views of the real world through a camera, and then combining with computer image technology to present on an opaque display. In addition, the main purpose of the video see-through method is to correctly display the pictures captured by two cameras on the display screen, align the pixels of the pictures, produce a stereoscopic effect, and align with the real world, so as to simulate the real world. However, the video see-through method has the problems that the two cameras cannot be completely aligned during installation, resulting in picture alignment error when displaying the binocular pictures, and the parallel alignment effect of the pictures and the line of sight is not considered.

[0084] To solve the above problems, the present application provides a video perspective method. The video perspective method of the present application obtains first image data captured by a first camera and second image data captured by a second camera, then performs stereo correction on the first image data and the second image data to obtain a first projection matrix corresponding to the first camera and a second projection matrix corresponding to the second camera, and generates third image data and fourth image data, wherein the third image data is the first image data after stereo correction, and the fourth image data is the second image data after stereo correction, then performs parallel correction on the third image data and the fourth image data based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix, to obtain first perspective image data and second perspective image data. Thus, the present application corrects the image obtained by the binocular camera, aligns the left and right binocular images, and makes the image parallel to the line of sight, thereby generating a stereo effect and improving user experience.

[0085] Please refer to Figure 2 , Figure 2 is a flowchart of the first embodiment of the video perspective method of the present application. It should be noted that although the logical order is shown in the flowchart, in some cases, the video perspective method of the present application can of course perform the steps shown or described in a different order than here.

[0086] In the first embodiment of the video perspective method of the present application, the video perspective method of the present application comprises:

[0087] Step S10: obtaining first image data captured by a first camera and second image data captured by a second camera;

[0088] In this embodiment, the terminal device obtains first image data captured by a first camera and second image data captured by a second camera.

[0089] For example, a VR head-mounted device is provided with one camera on the left and one camera on the right for capturing image data. Specifically, the first image data is captured by the left camera, and the second image data is captured by the right camera.

[0090] Step S20: performing stereo correction on the first image data to obtain third image data and a first projection matrix corresponding to the first camera;

[0091] In this embodiment, the terminal device performs stereo correction on the first image data captured by the first camera to obtain a first projection matrix corresponding to the first camera, and generates third image data after stereo correction.

[0092] Exemplarily, the image data collected by the left camera is Image0, the image data collected by the right camera is Image1, and the stereo rectification is performed on Image0 and Image1 to realize the alignment of two image frames. Specifically, the stereo rectification module in the OpenCV (Open Source Computer Vision Library) can be used to re-project the collected Image0, generate the first projection matrix Pl corresponding to the left camera, and obtain the image data RectifyImage0 of the left camera.

[0093] In step S30, the stereo rectification is performed on the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera.

[0094] In this embodiment, the terminal device performs the stereo rectification on the second image data collected by the second camera, obtains the second projection matrix corresponding to the second camera, and generates the fourth image data after the stereo rectification.

[0095] Exemplarily, the terminal device uses the stereo rectification module in the OpenCV to re-project the collected Image1, generates the second projection matrix Pr corresponding to the right camera, and obtains the image data RectifyImage1 of the right camera.

[0096] In step S40, the parallel rectification is performed on the third image data and the fourth image data based on the target rotation matrix corresponding to the first camera and the second camera, the first projection matrix, and the second projection matrix, to obtain first perspective image data and second perspective image data.

[0097] In this embodiment, the terminal device performs the parallel rectification on the third image data and the fourth image data based on the target rotation matrix corresponding to the first camera and the second camera, the first projection matrix corresponding to the first camera, and the second projection matrix corresponding to the second camera, to obtain the first perspective image data and the second perspective image data.

[0098] Exemplarily, since the camera on the VR headset has functions other than perspective, such as positioning, the camera implementation is not necessarily installed horizontally, and in order to correct the image to the effect of viewing straight ahead horizontally when the device is placed horizontally forward, the image captured by the camera needs to be corrected to the picture taken by the camera straight ahead at this time, i.e. RectifyImage0 and RectifyImage1 are further corrected. In mathematics, this operation is actually a rotation operation on the camera coordinate system, therefore, the key is to calculate a rotation matrix R for each of the left and right cameras, but since the two cameras have been adjusted to be aligned during the previous epipolar correction, i.e. the rotation matrix between them is already the identity matrix, so the rotation matrix used by the two cameras here is the same, i.e. the target rotation matrix. Using the target rotation matrix R and the first projection matrix Pl and the second projection matrix Pr obtained in the previous image correction process, RectifyImage0 and RectifyImage1 can be further corrected, i.e. parallel correction, and first perspective image data FinalImage0 and second perspective image data FinalImage1 can be obtained.

[0099] It should be noted that, as shown in Figure 3 The VR headset device is horizontally fixed with an inertial measurement unit for obtaining the user's line of sight information during use, and a camera is installed on each of the left and right sides of the VR headset device for collecting image data.

[0100] Further, in a possible embodiment, the video perspective method of the present application, the step S40 comprises:

[0101] Step S401, determining a first mapping matrix corresponding to the first camera based on the target rotation matrix and the first projection matrix;

[0102] Step S402, determining a second mapping matrix corresponding to the second camera based on the target rotation matrix and the second projection matrix;

[0103] Step S403, performing parallel correction on the third image data based on the first mapping matrix;

[0104] Step S404, performing parallel correction on the fourth image data based on the second mapping matrix.

[0105] In the embodiment, the terminal device determines a first mapping matrix corresponding to the first camera based on the target rotation matrix corresponding to the first camera and the first camera and a second mapping matrix corresponding to the second camera based on the target rotation matrix corresponding to the first camera and the second camera and the second projection matrix corresponding to the second camera, and performs parallel correction on the third image data generated after stereoscopic correction according to the first mapping matrix and performs parallel correction on the fourth image data generated after stereoscopic correction according to the second mapping matrix.

[0106] Exemplarily, the target rotation matrix R and the first projection matrix Pl and the second projection matrix Pr are transmitted to an image correction API (Application Programming Interface) of an open source image processing library OpenCV, two mapping matrices map0 and map1 are calculated by the image correction API, parallel correction is performed on the third image data RectifyImage0 according to the first mapping matrix map0, and parallel correction is performed on the fourth image data RectifyImage1 according to the second mapping matrix map1.

[0107] Further, in a feasible embodiment, the video perspective method of the present application can further include, after the step S40:

[0108] In step A, the first perspective image data and the second perspective image data are displayed on the VR display screen, wherein the first perspective image data is the third image data after parallel correction, and the second perspective image data is the fourth image data after parallel correction.

[0109] In the embodiment, the first perspective image data and the second perspective image data generated by the terminal device after parallel correction are displayed on the VR display screen.

[0110] Exemplarily, the third image data RectifyImage0 and the fourth image data RectifyImage1 are corrected into the first perspective image data FinalImage0 and the second perspective image data FinalImage1 by using the first mapping matrix map0 corresponding to the first camera and the second mapping matrix map1 corresponding to the second camera, and the first perspective image data FinalImage0 and the second perspective image data FinalImage1 are displayed on the VR display screen, so as to ensure that the picture seen by the user through the device is consistent with the picture effect seen by the left and right eyes of the user.

[0111] In the embodiment, the video perspective method of the present application acquires first image data collected by a first camera and second image data collected by a second camera through a terminal device; the terminal device performs stereo correction on the first image data collected by the first camera to obtain a first projection matrix corresponding to the first camera and generate third image data after stereo correction; the terminal device performs stereo correction on the second image data collected by the second camera to obtain a second projection matrix corresponding to the second camera and generate fourth image data after stereo correction; the terminal device determines a first mapping matrix corresponding to the first camera based on a target rotation matrix corresponding to the first camera and the second camera and the first projection matrix corresponding to the first camera, and determines a second mapping matrix corresponding to the second camera based on the target rotation matrix corresponding to the first camera and the second camera and the second projection matrix corresponding to the second camera, and performs parallel correction on the third image data generated after stereo correction according to the first mapping matrix and performs parallel correction on the fourth image data generated after stereo correction according to the second mapping matrix to obtain first perspective image data and second perspective image data; and the terminal device displays the first perspective image data and the second perspective image data generated after parallel correction on a VR display screen.

[0112] Thus, the embodiment of the present application acquires first image data photographed by a first camera and second image data photographed by a second camera, then performs stereo correction on the first image data and the second image data to obtain a first projection matrix corresponding to the first camera and a second projection matrix corresponding to the second camera and generate third image data and fourth image data, wherein the third image data is the first image data after stereo correction and the fourth image data is the second image data after stereo correction, then performs parallel correction on the third image data and the fourth image data based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix to obtain first perspective image data and second perspective image data. Thus, the present application corrects the image acquired by a binocular camera to align the left and right binocular pictures and make the pictures parallel to the line of sight, thereby generating a stereo effect and improving user experience.

[0113] Further, based on the first embodiment of the video perspective method of the present application, the second embodiment of the video perspective method of the present application is proposed.

[0114] In the embodiment, the dust removal control system of the present application further comprises a wharf central control of a wharf where the ship loader is located, and the ship loader and the belt scale are in communication connection with the wharf central control.

[0115] Before step S10, the video perspective method of the present application can further comprise:

[0116] Step B, calibrating the first camera and the second camera to obtain a first intrinsic parameter of the first camera, a second intrinsic parameter of the second camera, and a first extrinsic parameter between the first camera and the second camera.

[0117] In this embodiment, the terminal device calibrates the first camera and the second camera between obtaining the first image data and the second image data to obtain the intrinsic parameters of the two cameras respectively, and the extrinsic parameter between the two cameras, i.e., the first extrinsic parameter.

[0118] Exemplarily, the intrinsic parameters K and the distortion parameters D of the two cameras, and the extrinsic parameter between the two cameras are calibrated using an open source software package Kalibr, wherein the extrinsic parameter information includes a rotation matrix and a translation matrix between the left camera coordinate system and the right camera coordinate system.

[0119] Step C, determining a first rotation matrix corresponding to the first camera and a second rotation matrix corresponding to the second camera based on the first intrinsic parameter, the second intrinsic parameter, and the first extrinsic parameter.

[0120] In this embodiment, the terminal device determines a first rotation matrix corresponding to the first camera and a second rotation matrix corresponding to the second camera based on the first intrinsic parameter of the first camera, the second intrinsic parameter of the second camera, and the first extrinsic parameter between the two cameras.

[0121] Further, in a feasible embodiment, the above step S20 can include:

[0122] Step S201, performing stereo rectification on the first image data based on the first rotation matrix to obtain third image data and a first projection matrix corresponding to the first camera;

[0123] The above step S30 can include:

[0124] Step S301, performing stereo rectification on the second image data based on the second rotation matrix to obtain fourth image data and a second projection matrix corresponding to the second camera.

[0125] In this embodiment, the terminal device performs stereo rectification on the first image data based on the first rotation matrix, and performs stereo rectification on the second image data based on the second rotation matrix.

[0126] Exemplarily, to realize the stereoscopic correction of the left and right camera images, a stereoscopic correction module in an open source computer vision library OpenCV can be used in software to perform polar line alignment, i.e. row alignment, between the two images according to the camera internal parameters, the distortion parameters, and the external parameters of the two cameras. After the polar line alignment, the two cameras are mathematically aligned, i.e. aligned to the same observation plane, and the pixels are strictly row-aligned, and the imaging effect is as shown in Figure 4 wherein Pl is a projection matrix of the left camera after the stereoscopic correction, i.e. a first projection matrix, and Pr is a projection matrix of the right camera after the stereoscopic correction, i.e. a second projection matrix.

[0127] In the embodiment, the video see-through method of the present application calibrates the first camera and the second camera to obtain the internal parameters of the two cameras and the external parameters, i.e. the first external parameters, between the two cameras, between the first image data and the second image data; determines the first rotation matrix corresponding to the first camera and the second rotation matrix corresponding to the second camera based on the first internal parameters of the first camera, the second internal parameters of the second camera, and the first external parameters between the two cameras, and then performs stereoscopic correction on the first image data based on the first rotation matrix and performs stereoscopic correction on the second image data based on the second rotation matrix.

[0128] In this way, the stereoscopic correction module in the open source computer vision library OpenCV is used to perform stereoscopic correction on the images collected by the left and right cameras, row alignment of the left and right pictures is realized, and thus the user can have a stereoscopic effect when watching.

[0129] Further, based on the first embodiment and / or the second embodiment of the video see-through method of the present application, a third embodiment of the video see-through method of the present application is proposed.

[0130] In the embodiment, the video see-through method of the present application can further include:

[0131] Step D, calibrating the first camera and the inertial measurement unit to obtain second external parameters between the first camera and the inertial measurement unit;

[0132] Step E, determining a target rotation matrix corresponding to the first camera and the second camera based on the second external parameters and the first rotation matrix.

[0133] In the embodiment, the terminal device calibrates the first camera and the inertial measurement unit to obtain second external parameters between the first camera and the inertial measurement unit, and then determines a target rotation matrix corresponding to the first camera and the second camera according to the second external parameters and the first rotation matrix generated during the stereoscopic correction.

[0134] Exemplarily, a target rotation matrix between the left camera and the IMU (Inertial measurement unit) is R, and it needs to be noted that the coordinate system orientation of the IMU and the coordinate system orientation of the two cameras after stereo rectification may not be consistent, thereby leading to incorrect pictures. Therefore, based on the rotation matrix in the external parameter between the left camera coordinate system and the IMU coordinate system and the first rotation matrix generated during stereo rectification, the target rotation matrix R is calculated, so that the camera coordinate system and the IMU coordinate system can be parallel after the camera coordinate system is rotated according to the target rotation matrix R.

[0135] Exemplarily, as shown in Figure 8 the left camera camera0, the first image data Image0 collected by the left camera, and the first image data Image1 collected by the right camera camera1 are acquired, the cameras 0 and 1 are calibrated to obtain the internal parameters and distortion parameters of the camera 0 and the camera 1, and the external parameters between the camera 0 and the camera 1, the rotation matrix of the camera 0 and the camera 1 is calculated based on the parameters obtained by the calibration, and the stereo rectification is performed on Image0 and Image1 according to the rotation matrix of the two cameras respectively to generate the projection matrix P corresponding to the two cameras respectively, wherein the projection matrix of the camera 0 is Pl, and the projection matrix of the camera 1 is Pr. In addition, the images obtained after the stereo rectification are the third image data RectifyImage0 and the fourth image data RectifyImage1. Then, the external parameters between the camera 0 and the IMU are determined, so as to determine the target rotation matrix R according to the rotation matrix in the external parameters and the rotation matrix of the camera 0 and the camera 1 obtained by calibrating the two cameras, and the parallel rectification is performed on RectifyImage0 and RectifyImage1 according to the target rotation matrix and the projection matrix P corresponding to the two cameras respectively to obtain the first perspective image data FinalImage0 and the second perspective image data FinalImage1.

[0136] Further, in a possible embodiment, the video perspective method of the present application, the step S302 can include:

[0137] Step F, determining the first coordinate system orientation of the inertial measurement unit according to the second external parameter;

[0138] Step G, determining the second coordinate system orientation of the first camera according to the first rotation matrix;

[0139] Step H, determining the target rotation matrix corresponding to the first camera and the second camera according to the first coordinate system orientation and the second coordinate system orientation.

[0140] In this embodiment, the rotation matrix in the extrinsic parameters between the first camera and the inertial measurement unit of the terminal device determines the coordinate system orientation of the inertial measurement unit, i.e., the first coordinate system orientation. Based on the first rotation matrix generated during stereo calibration, the coordinate system orientation of the first camera is determined, i.e., the second coordinate system orientation. Then, the target rotation matrix corresponding to the first camera and the second camera is determined based on the first coordinate system orientation and the second coordinate system orientation.

[0141] It should be noted that after stereo correction, the image data acquired by the two cameras are aligned in rows. Therefore, the rotation matrix between the two cameras is an identity matrix, meaning that the coordinate systems of the two cameras are oriented in the same direction.

[0142] For example, the camera coordinate system is oriented as shown in Figure 5, and the IMU coordinate system is oriented as shown in Figure 6. Figure 6 As shown, first, apply two rotations to the camera coordinate system: a negative 90-degree rotation around the Z-axis and a 180-degree rotation around the Y-axis, so that the camera coordinate system axes are parallel to the IMU coordinate axes, that is, make the camera coordinate system as shown. Figure 7 As shown, the camera coordinate system starts from... Figure 5 The coordinate system shown is rotated to Figure 7 The formula for the rotation amount of the coordinate system orientation shown is R = Rz90_inv * Ry180_inv * Rb_inv * R0_inv, where R is the target rotation matrix between the left camera and the IMU; R0_inv is the inverse of the rotation matrix required for the left camera in epipolar alignment; Rz90_inv is the inverse of the rotation matrix corresponding to a negative 90-degree rotation around the Z-axis; Ry180_inv is the inverse of the rotation matrix corresponding to a 180-degree rotation around the Y-axis; and Rb_inv is the inverse of the rotation matrix between the IMU and the camera. Specifically, the rotation matrix in the extrinsic parameters between the left camera and the IMU is Rb.

[0143] In this embodiment, the video perspective method of the present invention calibrates the first camera and the inertial measurement unit (IMU) using a terminal device to obtain a second extrinsic parameter between the first camera and the IMU. Then, based on the second extrinsic parameter and the first rotation matrix generated during stereo calibration, the target rotation matrix corresponding to the first camera and the second camera is determined. The rotation matrix in the extrinsic parameter between the first camera and the IMU of the terminal device determines the coordinate system orientation of the IMU, i.e., the first coordinate system orientation. Based on the first rotation matrix generated during stereo calibration, the coordinate system orientation of the first camera, i.e., the second coordinate system orientation, is determined. Then, the target rotation matrix corresponding to the first camera and the second camera is determined based on the first coordinate system orientation and the second coordinate system orientation.

[0144] Therefore, by calculating the rotation matrix of the camera coordinate system, the image data after the stereo correction is further corrected, the picture is aligned with the user's line of sight, and the use experience of the user is improved.

[0145] In addition, the embodiment of the present application also provides a video perspective device.

[0146] Please refer to Figure 9 , Figure 9 The function module schematic diagram of an embodiment of the video perspective device of the present application is shown in Figure 9 The video perspective device of the present application comprises:

[0147] The image acquisition module 10 is configured to acquire first image data collected by a first camera and second image data collected by a second camera.

[0148] The first stereo correction module 20 is configured to perform stereo correction on the first image data to obtain third image data and a first projection matrix corresponding to the first camera.

[0149] The second stereo correction module 30 is configured to perform stereo correction on the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera.

[0150] The parallel correction module 40 is configured to perform parallel correction on the third image data and the fourth image data based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix and the second projection matrix, to obtain first perspective image data and second perspective image data.

[0151] Optionally, the video perspective device of the present application further comprises:

[0152] The first calibration module is configured to calibrate the first camera and the second camera to obtain a first intrinsic parameter of the first camera, a second intrinsic parameter of the second camera and a first extrinsic parameter between the first camera and the second camera.

[0153] The rotation matrix module is configured to determine a first rotation matrix corresponding to the first camera and a second rotation matrix corresponding to the second camera based on the first intrinsic parameter, the second intrinsic parameter and the first extrinsic parameter.

[0154] Optionally, the first stereo correction module 20 is further configured to perform stereo correction on the first image data based on the first rotation matrix to obtain the third image data and the first projection matrix corresponding to the first camera; and the second stereo correction module 30 is further configured to perform stereo correction on the second image data based on the second rotation matrix to obtain the fourth image data and the second projection matrix corresponding to the second camera.

[0155] Optionally, the video see-through device further comprises:

[0156] a second calibration module, configured to calibrate the first camera and the inertial measurement unit to obtain a second extrinsic parameter between the first camera and the inertial measurement unit;

[0157] a target rotation matrix unit, configured to determine a target rotation matrix corresponding to the first camera and the second camera based on the second extrinsic parameter and the first rotation matrix.

[0158] Optionally, the target rotation matrix unit is further configured to determine a first coordinate system orientation of the inertial measurement unit according to the second extrinsic parameter, determine a second coordinate system orientation of the first camera according to the first rotation matrix, and determine the target rotation matrix corresponding to the first camera and the second camera according to the first coordinate system orientation and the second coordinate system orientation.

[0159] Optionally, the parallel correction module 40 comprises

[0160] a first mapping matrix unit, configured to determine a first mapping matrix corresponding to the first camera based on the target rotation matrix and the first projection matrix;

[0161] a second mapping matrix unit, configured to determine a second mapping matrix corresponding to the second camera based on the target rotation matrix and the second projection matrix;

[0162] a first parallel correction unit, configured to perform parallel correction on the third image data based on the first mapping matrix;

[0163] a second parallel correction unit, configured to perform parallel correction on the fourth image data based on the second mapping matrix.

[0164] Optionally, the video see-through device further comprises:

[0165] a display module, configured to display the first perspective image data and the second perspective image data on a VR display screen, wherein the first perspective image data is the third image data after parallel correction, and the second perspective image data is the fourth image data after parallel correction.

[0166] The application further provides a computer storage medium, which stores a video see-through program, and the video see-through program, when executed by a processor, implements the steps of the video see-through program method according to any one of the above embodiments.

[0167] The specific embodiments of the computer storage medium of the application are basically the same as the above-mentioned embodiments of the video see-through program method of the application, and will not be repeated here.

[0168] The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the video see-through method according to the present application as described in any of the above embodiments, which will not be repeated here.

[0169] It should be noted that, in the present document, the terms "comprising", "containing" or any other similar term are intended to encompass non-exclusive inclusions, such that a process, a method, an article or a system that comprises a list of elements does not only include those elements, but can also include other elements not explicitly listed, or can also include elements inherent to such a process, method, article or system. Without further limitations, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or system that includes the element.

[0170] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0171] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software and a necessary general hardware platform, and of course, they can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) as described above, and includes a number of instructions for making a terminal device (which can be a TWS earphone or the like) execute the methods described in the various embodiments of the present application.

[0172] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method of video see-through, characterized by, The video perspective method comprises: obtaining first image data collected by a first camera and second image data collected by a second camera, wherein the first camera and the second camera are cameras on a VR head-mounted device; stereo correcting the first image data to obtain third image data and a first projection matrix corresponding to the first camera; stereo correcting the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera; parallel correcting the third image data and the fourth image data based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix, and the second projection matrix to obtain first perspective image data and second perspective image data; The step of parallel correcting the third image data and the fourth image data based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix, and the second projection matrix comprises: determining a first mapping matrix corresponding to the first camera based on the target rotation matrix and the first projection matrix; determining a second mapping matrix corresponding to the second camera based on the target rotation matrix and the second projection matrix; parallel correcting the third image data based on the first mapping matrix; parallel correcting the fourth image data based on the second mapping matrix.

2. The video see-through method of claim 1, wherein, Before the step of obtaining first image data collected by a first camera and second image data collected by a second camera, the method further comprises: calibrating the first camera and the second camera to obtain a first intrinsic parameter of the first camera, a second intrinsic parameter of the second camera, and a first extrinsic parameter between the first camera and the second camera; determining a first rotation matrix corresponding to the first camera and a second rotation matrix corresponding to the second camera based on the first intrinsic parameter, the second intrinsic parameter, and the first extrinsic parameter.

3. The video see-through method of claim 2, wherein, The step of stereo correcting the first image data to obtain third image data and a first projection matrix corresponding to the first camera comprises: stereo correcting the first image data based on the first rotation matrix to obtain third image data and a first projection matrix corresponding to the first camera; The step of stereo correcting the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera comprises: stereo correcting the second image data based on the second rotation matrix to obtain fourth image data and a second projection matrix corresponding to the second camera.

4. The video see-through method of claim 3, wherein, The method further comprises: calibrating the first camera and an inertial measurement unit to obtain a second extrinsic parameter between the first camera and the inertial measurement unit; determining a target rotation matrix corresponding to the first camera and the second camera based on the second extrinsic parameter and the first rotation matrix.

5. The video see-through method of claim 4, wherein, The step of determining a target rotation matrix corresponding to the first camera and the second camera based on the second extrinsic parameter and the first rotation matrix comprises: determine a first coordinate system orientation of the inertial measurement unit according to the second extrinsic parameter; determine a second coordinate system orientation of the first camera according to the first rotation matrix; determine a target rotation matrix corresponding to the first camera and the second camera according to the first coordinate system orientation and the second coordinate system orientation.

6. The video see-through method of claim 1, wherein, After the step of obtaining the first perspective image data and the second perspective image data, the method further comprises: display the first perspective image data and the second perspective image data on a VR display screen, wherein the first perspective image data is the third image data after parallel correction, and the second perspective image data is the fourth image data after parallel correction.

7. A video see-through device, characterized by, The video perspective device comprises: an image acquisition module configured to acquire first image data collected by a first camera and second image data collected by a second camera, wherein the first camera and the second camera are cameras on a VR head-mounted device; a first stereo correction module configured to perform stereo correction on the first image data to obtain third image data and a first projection matrix corresponding to the first camera; a second stereo correction module configured to perform stereo correction on the second image data to obtain fourth image data and a second projection matrix corresponding to the second camera; a parallel correction module configured to perform parallel correction on the third image data and the fourth image data based on a target rotation matrix corresponding to the first camera and the second camera, the first projection matrix, and the second projection matrix, to obtain first perspective image data and second perspective image data; determine a first mapping matrix corresponding to the first camera based on the target rotation matrix and the first projection matrix; determine a second mapping matrix corresponding to the second camera based on the target rotation matrix and the second projection matrix; perform parallel correction on the third image data based on the first mapping matrix; and perform parallel correction on the fourth image data based on the second mapping matrix.

8. A terminal device, comprising: The terminal device comprises a memory, a processor, and a video perspective program stored on the memory and executable on the processor, and the video perspective program, when executed by the processor, implements the steps of the video perspective method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a video perspective program, and the video perspective program, when executed by the processor, implements the steps of the video perspective method according to any one of claims 1 to 6.

Images