Multi-channel curved surface projection correction method, device, equipment, medium and product

Abstract

The invention discloses a multi-channel curved surface projection correction method and device, equipment, a medium and a product, and relates to the field of projection correction, and the method comprises the steps: carrying out the mapping of a projection picture shot by a left channel camera after the projection through a left channel according to a left mapping matrix; according to the right mapping matrix, mapping a projection picture shot by a right channel camera after projection through the right channel; according to the middle mapping matrix, mapping a projection picture shot by the middle channel camera after being projected through the middle channel; and mapping the mapped image by using a comprehensive mapping matrix, performing inverse mapping on the mapped image to obtain an inverse mapping relationship, and performing pre-distortion and projection on the to-be-corrected image according to the inverse mapping relationship to complete correction of the to-be-corrected image. Correction can be carried out under the condition that the viewpoint camera is shielded by a mechanical structure and a picture is missing without depending on position information between the camera and the projector.

Description

Technical Field

[0001] The present application relates to the field of projection correction, and in particular to a multi-channel curved surface projection correction method, device, equipment, medium and product. Background Art

[0002] In most current flight simulation equipment, visual imaging is performed by projection. Projection on the actual screen is real image imaging, and after multiple optical processes such as lenses, it becomes virtual image imaging. In order to achieve realistic effects during simulation, large field of view and multi-channel projection are used.

[0003] Before image stitching, the image must be corrected. The common correction methods in related technologies can be roughly divided into two categories:

[0004] The first is the manual correction method. This method requires setting up a total station near the viewpoint designed for the equipment cabin, and manually finding the angle of each curtain calibration point relative to the viewpoint. The entire surface is inferred based on the position angle of all points relative to the viewpoint. The correction result of this method depends largely on people's subjective feelings, resulting in low accuracy. When the flight simulation platform is running, in order to simulate the feeling of weightlessness and overweight, the entire simulation cabin will often move. The fixedly installed projector may have deviations due to shaking, causing the projection splicing strips of adjacent channels to produce ghosting. Once the ghosting affects the normal training of the pilot, it needs to be re-stitched, and then the image needs to be re-calibrated. However, due to manual operation, each calibration takes a long time and has high operating costs, which greatly delays the normal training plan. In addition, due to differences in technology and processes for some models of flight simulators: the diffuse reflection curtain of the virtual image imaging system is marked with fluorescent calibration points according to longitude and latitude before leaving the factory, but the diffuse reflection curtain of the real image imaging will not mark the field of view angle. It is necessary to manually mark the longitude and latitude of the curtain before calibration, which is very time-consuming and inaccurate.

[0005] The second is automatic program correction, which collects the calibration point information on the projection screen and automatically calibrates it through camera acquisition to obtain the correction matrix. This method is limited by the camera shooting accuracy and it is difficult to strictly align the calibration grid. As for the geometric correction method using the camera, more importantly, there is a problem of equipment occlusion inside the cabin of some flight simulators, which makes it impossible for the calibration camera to fully capture all the calibration points at the viewpoint. Due to the missing of some calibration points, the subsequent stitching work cannot be carried out normally.

[0006] Therefore, a multi-channel curved surface projection correction method is needed that does not require calibration points on the screen, does not rely on the position information between the camera and the projector, and can perform correction when the viewpoint camera is blocked by a mechanical structure and causes image loss. Summary of the invention

[0007] The purpose of this application is to provide a multi-channel curved surface projection correction method, device, equipment, medium and product, which does not require calibration points on the screen, does not rely on position information between the camera and the projector, and can perform correction when the viewpoint camera is blocked by a mechanical structure, resulting in missing images.

[0008] To achieve the above objectives, this application provides the following solutions:

[0009] In a first aspect, the present application provides a multi-channel curved surface projection correction method, comprising:

[0010] After projecting the reference images of each position onto the screen through each channel, the viewpoint camera and each channel camera take projection images; the channels include the left channel, the middle channel and the right channel, and one channel corresponds to one channel camera;

[0011] After each position reference picture is projected onto the screen through the left channel and the middle channel respectively, the projection picture taken by the channel camera corresponding to the left channel is mapped according to the left mapping matrix, so as to obtain a mapping image of the projection picture taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; the left mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the left channel and the projection picture taken by the viewpoint camera after each position reference picture is projected onto the screen through the left channel and the middle channel respectively;

[0012] After each position reference picture is projected onto the screen through the middle channel and the right channel, the projection picture taken by the channel camera corresponding to the right channel is mapped according to the right mapping matrix, so as to obtain a mapping image of the projection picture taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively; the right mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the right channel and the projection picture taken by the viewpoint camera after each position reference picture is projected onto the screen through the middle channel and the right channel respectively;

[0013] A first completed image is obtained according to a projection image taken by a rear viewpoint camera after the first position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the first position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the first position reference image is projected onto the screen through the middle channel;

[0014] A second completed image is obtained according to a projection image taken by a rear viewpoint camera after the second position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the second position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the second position reference image is projected onto the screen through the middle channel;

[0015] After each position reference picture is projected onto the screen through the intermediate channel, the projection picture taken by the channel camera corresponding to the intermediate channel is mapped according to the intermediate mapping matrix, so as to obtain a mapping image of the projection picture taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel; the intermediate mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel, the first complement image and the second complement image;

[0016] A comprehensive mapping matrix is ​​used to map the mapping images of the projection images taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference picture is projected onto the screen through the middle channel, and an interpolation inverse mapping operation is performed on the mapped images to obtain an inverse mapping relationship corresponding to the left channel, an inverse mapping relationship corresponding to the right channel, and an inverse mapping relationship corresponding to the middle channel; the comprehensive mapping matrix is ​​obtained according to the mapping images of the projection images taken by the channel camera corresponding to the left channel after the first position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after the first position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after the first position reference picture is projected onto the screen through the middle channel;

[0017] The image to be corrected is pre-distorted by using the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel, and the pre-distorted image is projected to complete the correction of the image to be corrected.

[0018] In a second aspect, the present application provides a multi-channel curved surface projection stitching correction device, comprising:

[0019] The acquisition module is used to project the reference images of each position onto the screen through each channel, and then the viewpoint camera and each channel camera take the projection images; the channels include the left channel, the middle channel and the right channel, and one channel corresponds to one channel camera;

[0020] A first mapping module is used to map the projection pictures taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively according to the left mapping matrix, so as to obtain a mapping image of the projection pictures taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; the left mapping matrix is ​​obtained according to the projection pictures taken by the channel camera corresponding to the left channel and the projection pictures taken by the viewpoint camera after each position reference picture is projected onto the screen through the left channel and the middle channel respectively;

[0021] A second mapping module is used to map the projection pictures taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively according to the right mapping matrix, so as to obtain a mapping image of the projection pictures taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively; the right mapping matrix is ​​obtained according to the projection pictures taken by the channel camera corresponding to the right channel and the projection pictures taken by the viewpoint camera after each position reference picture is projected onto the screen through the middle channel and the right channel respectively;

[0022] A first completion module, for obtaining a first completion image according to a projection image taken by a rear viewpoint camera after the first position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the first position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the first position reference image is projected onto the screen through the middle channel;

[0023] A second completion module is used to obtain a second completion image according to a projection image taken by a rear viewpoint camera after the second position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the second position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the second position reference image is projected onto the screen through the middle channel;

[0024] A third mapping module is used to map the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel according to the intermediate mapping matrix, so as to obtain a mapping image of the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel; the intermediate mapping matrix is ​​obtained according to the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel, the first complement image and the second complement image;

[0025] An inverse mapping module is used to use a comprehensive mapping matrix to map the mapping images of the projection images taken by the channel camera corresponding to the left channel after each position reference image is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after each position reference image is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference image is projected onto the screen through the middle channel, and perform interpolation inverse mapping operations on the mapped images to obtain the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel; the comprehensive mapping matrix is ​​obtained according to the mapping images of the projection images taken by the channel camera corresponding to the left channel after the first position reference image is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after the first position reference image is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after the first position reference image is projected onto the screen through the middle channel;

[0026] The correction module is used to pre-distort the image to be corrected by using the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel and the inverse mapping relationship corresponding to the middle channel, and project the pre-distorted image to complete the correction of the image to be corrected.

[0027] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned multi-channel curved surface projection correction method.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned multi-channel curved surface projection correction method when executed by a processor.

[0029] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned multi-channel curved surface projection correction method when executed by a processor.

[0030] According to the specific embodiments provided in this application, this application has the following technical effects:

[0031] The present application provides a multi-channel curved surface projection correction method, device, equipment, medium and product. After each position reference picture is projected onto the screen through the left channel and the middle channel respectively, the projection picture taken by the channel camera corresponding to the left channel is mapped according to the left mapping matrix; after each position reference picture is projected onto the screen through the middle channel and the right channel respectively, the projection picture taken by the channel camera corresponding to the right channel is mapped according to the right mapping matrix; a first completion image is obtained according to the mapping image of the projection picture taken by the rear viewpoint camera after the first position reference picture is projected onto the screen through the middle channel, the projection picture taken by the channel camera corresponding to the left channel after the first position reference picture is projected onto the screen through the middle channel, and the projection picture taken by the channel camera corresponding to the right channel after the first position reference picture is projected onto the screen through the middle channel. image; obtain a second complement image according to the projection image taken by the viewpoint camera after the second position reference image is projected onto the screen through the middle channel, the mapping image of the projection image taken by the channel camera corresponding to the left channel after the second position reference image is projected onto the screen through the middle channel, and the mapping image of the projection image taken by the channel camera corresponding to the right channel after the second position reference image is projected onto the screen through the middle channel; map the projection images taken by the channel camera corresponding to the middle channel after each position reference image is projected onto the screen through the middle channel respectively according to the intermediate mapping matrix; complement the occluded part of the occluded image taken by the view camera according to the image taken by the channel camera, so that correction can be performed when the viewpoint camera is blocked by the mechanical structure and the picture is missing, and no calibration points are required on the screen, and it does not rely on the position information between the camera and the projector. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of a flow chart of a multi-channel curved surface projection correction method provided in one embodiment of the present application;

[0034] Figure 2 A schematic diagram of the camera position configuration for the calibration system;

[0035] Figure 3 Reference image for the first position;

[0036] Figure 4 Reference image for the second position;

[0037] Figure 5 a in the figure is the first position reference image projected by channel A0 taken by the CamA0 camera;

[0038] Figure 5 b in the figure is the first position reference image projected by channel A0 taken by camera CamA1;

[0039] Figure 5 c in the figure is the first position reference image projected by channel A0 taken by the CamA2 camera;

[0040] Figure 5 d in the figure is the first position reference image projected by channel A1 taken by camera CamA0;

[0041] Figure 5 e in the figure is the first position reference image projected by channel A1 taken by the CamA1 camera;

[0042] Figure 5 f in the figure is the first position reference image projected by channel A1 taken by the CamA2 camera;

[0043] Figure 5 g in the figure is the first position reference image projected by channel A2 taken by the CamA0 camera;

[0044] Figure 5 h in the figure is the first position reference image projected by channel A2 taken by the CamA1 camera;

[0045] Figure 5 The i in the figure is the first position reference image projected by channel A2 taken by the CamA2 camera;

[0046] Figure 6 a in the figure is the second position reference image projected by channel A0 taken by the CamA0 camera;

[0047] Figure 6 b in the figure is the second position reference image projected by channel A0 taken by the CamA1 camera;

[0048] Figure 6 c in the figure is the second position reference image projected by channel A0 taken by the CamA2 camera;

[0049] Figure 6 d in the figure is the second position reference image projected by channel A1 taken by camera CamA0;

[0050] Figure 6e in the figure is the second position reference image projected by channel A1 taken by the CamA1 camera;

[0051] Figure 6 f in the figure is the second position reference image projected by channel A1 taken by the CamA2 camera;

[0052] Figure 6 g in the figure is the second position reference image projected by channel A2 taken by the CamA0 camera;

[0053] Figure 6 h in the figure is the second position reference image projected by channel A2 taken by the CamA1 camera;

[0054] Figure 6 The i in the figure is the second position reference image projected by channel A2 taken by the CamA2 camera;

[0055] Figure 7 a in the figure is the first position reference image projected by channel A0 taken by the CamS camera;

[0056] Figure 7 b in the figure is the first position reference image projected by channel A1 taken by the CamS camera;

[0057] Figure 7 c in the figure is the first position reference image projected by channel A2 taken by the CamS camera;

[0058] Figure 7 d in the figure is the second position reference image projected by channel A0 taken by the CamS camera;

[0059] Figure 7 e in the figure is the second position reference image projected by channel A1 taken by the CamS camera;

[0060] Figure 7 f in the figure is the second position reference image projected by channel A2 taken by the CamS camera;

[0061] Figure 8 a in the figure is the value after being mapped by the left mapping matrix. Figure 5 a in;

[0062] Figure 8 b in Figure 8 Schematic diagram of the addition of d and f in 7 and b in 7;

[0063] Figure 8 The c in the figure is the value after being mapped by the mapping matrix on the right. Figure 5 i in;

[0064] Figure 8 The d in the figure is the value after being mapped by the left mapping matrix. Figure 5 d in;

[0065] Figure 8 The e in the figure is the value after being mapped by the intermediate mapping matrix. Figure 5 e in;

[0066] Figure 8 The f in the figure is the value after being mapped by the mapping matrix on the right. Figure 5 f in;

[0067] Figure 8 The g in Figure 8 a in the figure is flattened and expanded;

[0068] Figure 8 The h in Figure 8 The e in the figure is flattened and expanded;

[0069] Figure 8 The i in Figure 8 The c in the figure is flattened and expanded;

[0070] Fig. 9 a in the figure is the value after being mapped by the left mapping matrix. Figure 6 a in;

[0071] Fig. 9 b in Fig. 9 Schematic diagram of the addition of d and f in 7 and e in 7;

[0072] Fig. 9 The c in the figure is the value after being mapped by the mapping matrix on the right. Figure 6 i in;

[0073] Fig. 9 The d in the figure is the value after being mapped by the left mapping matrix. Figure 6 d in;

[0074] Fig. 9 The e in the figure is the value after being mapped by the intermediate mapping matrix. Figure 6 e in;

[0075] Fig. 9 The f in the figure is the value after being mapped by the mapping matrix on the right. Figure 6 f in;

[0076] Fig. 9 The g in Fig. 9 a in the figure is flattened and expanded;

[0077] Fig. 9 The h in Fig. 9 The e in the figure is flattened and expanded;

[0078] Fig. 9 The i in Fig. 9 The c in the figure is flattened and expanded;

[0079] Fig.10 For the general Figure 8 The a, c and e in the figure are added to obtain the composite image;

[0080] Fig.11 Schematic diagram of the inverse mapping algorithm;

[0081] Fig.12 is the image to be corrected;

[0082] Fig.13 a in the figure is the pre-distortion image formed by pre-distorting the image to be corrected using the corresponding M_inv in the A0 channel;

[0083] Fig.13 b in the figure is the pre-distortion image formed by pre-distorting the image to be corrected using the corresponding M_inv in the A1 channel;

[0084] Fig.13 Where c is the pre-distortion image formed by pre-distorting the image to be corrected using the corresponding M_inv in the A2 channel;

[0085] Fig.14 The a in the Fig.13 a in the image is projected onto the screen through the A0 channel and part of Fig.13 The actual picture of b being projected onto the screen through channel A1;

[0086] Fig.14 The b in the Fig.13 The a in the image is projected onto the screen through the A0 channel and Fig.13 b is a real shot of the overlapping area of ​​the two channels projected onto the screen through channel A1;

[0087] Fig.14 The c in the Fig.13 b is projected onto the screen through channel A1 and Fig.13 c is a real shot of the overlapping area of ​​the two channels projected onto the screen through channel A2;

[0088] Fig.14 The d in the figure is the part Fig.13 b is projected onto the screen through channel A1 and Fig.13 Actual picture of c being projected onto the screen through the A2 channel. DETAILED DESCRIPTION

[0089] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0090] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0091] Related technology An automatic surface projection correction and stitching method comprises the following steps: using a network camera module to communicate with the camera and using file mapping to read the image in the main program. Manually selecting the approximate area of ​​the screen calibration point and having the program accurately calculate the sub-pixel coordinates. The projector projects an elliptical curve, the camera collects the image on the screen, and the ellipse parameters are adjusted according to the iteration strategy for fitting. The Apriltag image after correction projected by the middle channel is used to calculate the stitching anchor point of the stitching band. The splicing band fusion is realized. The splicing band color transition exists. The following problems exist:

[0092] 1. Although the stitching correction time is much shorter than manual time, the single-channel scene correction time still takes 1-2 hours.

[0093] 2. Using the projected elliptic curve to fit the curtain calibration points is theoretically a local convergence model, but in actual use there are cases of fitting failure or oscillation.

[0094] 3. The calibration splicing model can only be applied to spherical and quasi-spherical screens and cannot adapt to special-shaped screens.

[0095] 4. The stitching band fusion method is only applicable to horizontal stitching and cannot meet the stitching requirements between multiple rows and columns of projections.

[0096] 5. The system strategy and communication module have low degrees of freedom and cannot meet customized correction strategies.

[0097] Related technology A method, system, device and medium for stitching and correcting arbitrary curved surface projection. The steps after upgrading the above method are: obtaining the image taken by the camera. Identification and detection of calibration points on the screen. Coarse and fine calibration of calibration points based on row and column scanning. Stitching of stitching strips in complex situations. Color transition of stitching strips. The problems in the above automatic curved surface projection correction stitching method can be solved, but the above two methods both rely on the existence of calibration points on the imaging screen, which are only available in virtual imaging systems. If they need to be used in real imaging systems, the screen needs to be calibrated additionally. There are the following problems:

[0098] 1. The calibration effect is good only in the virtual image system. The real image system needs to manually mark the calibration points on the screen, which will increase the calibration error.

[0099] 2. Although the calibration time has been shortened from 3-4 hours to 30 minutes, there is still room for improvement.

[0100] Related technologies such as "A Novel Multi-Projection Correction Method Based on Binocular Vision" and "Multiple Projector Camera Calibration by Fiducial Marker Detection" proposed a method for calibration without the need for calibration points on the screen. The principle is briefly described as follows: several cameras are set up at the viewpoint to collect information at the projection point of the screen, and the pixel position of the corrected projection image corresponding to the image before correction is determined based on the projected markers, so as to perform interpolation and smoothing operations. Although the limitation of the calibration points of the screen is eliminated, the camera must be placed at the designed viewpoint and there must be no obstruction. However, due to the mechanical structure limitations of the flight simulator, there will inevitably be mechanical structures such as instruments blocking the space from the viewpoint to the screen, resulting in the loss of some information in the projected image, so this method cannot be simply used.

[0101] Based on this, in an exemplary embodiment, Figure 1 As shown, a multi-channel curved surface projection correction method is provided, which can solve the above problems. The multi-channel curved surface projection correction method provided in this embodiment includes the following steps, wherein:

[0102] Step 201: After projecting each position reference image onto the screen through each channel, the viewpoint camera and each channel camera take projection images. The channels include the left channel, the middle channel and the right channel, and one channel corresponds to one channel camera.

[0103] Step 202: After each position reference picture is projected onto the screen through the left channel and the middle channel, the projection picture taken by the channel camera corresponding to the left channel is mapped according to the left mapping matrix to obtain a mapping image of the projection picture taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel. The left mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the left channel and the projection picture taken by the viewpoint camera after each position reference picture is projected onto the screen through the left channel and the middle channel.

[0104] Step 203: After each position reference picture is projected onto the screen through the middle channel and the right channel, the projection picture taken by the channel camera corresponding to the right channel is mapped according to the right mapping matrix to obtain a mapping image of the projection picture taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel. The right mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the right channel and the projection picture taken by the viewpoint camera after each position reference picture is projected onto the screen through the middle channel and the right channel.

[0105] Step 204: Obtain a first complement image based on the projection image captured by the rear view camera after the first position reference image is projected onto the screen through the middle channel, the mapping image of the projection image captured by the channel camera corresponding to the left channel after the first position reference image is projected onto the screen through the middle channel, and the mapping image of the projection image captured by the channel camera corresponding to the right channel after the first position reference image is projected onto the screen through the middle channel. Specifically, the three images are added to obtain the first complement image.

[0106] Step 205: Obtain a second complement image based on the projection image taken by the viewpoint camera after the second position reference image is projected onto the screen through the middle channel, the mapping image of the projection image taken by the channel camera corresponding to the left channel after the second position reference image is projected onto the screen through the middle channel, and the mapping image of the projection image taken by the channel camera corresponding to the right channel after the second position reference image is projected onto the screen through the middle channel. Specifically, the three images are added together to obtain the second complement image. The above two complement images complement the missing part of the projection image taken by the viewpoint camera due to occlusion.

[0107] Step 206: After each position reference picture is projected onto the screen through the intermediate channel, the projection picture taken by the channel camera corresponding to the intermediate channel is mapped according to the intermediate mapping matrix to obtain a mapping image of the projection picture taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel. The intermediate mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel, the first complementary image, and the second complementary image.

[0108] Step 207: Use the comprehensive mapping matrix to map the mapping images of the projection images taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference picture is projected onto the screen through the middle channel, and perform interpolation inverse mapping operations on the mapped images to obtain the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel. The comprehensive mapping matrix is ​​obtained according to the mapping images of the projection images taken by the channel camera corresponding to the left channel after the first position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after the first position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after the first position reference picture is projected onto the screen through the middle channel.

[0109] Step 208: pre-distort the image to be corrected using the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel, and project the pre-distorted image to complete the correction of the image to be corrected.

[0110] The implementation of the above steps 201 to 208 does not require calibration points on the screen, does not rely on position information between the camera and the projector, and can perform correction when the viewpoint camera is blocked by a mechanical structure, resulting in missing images.

[0111] In another exemplary embodiment of the present application, after each position reference picture is projected onto the screen through the left channel and the middle channel respectively, the projection picture taken by the channel camera corresponding to the left channel is mapped according to the left mapping matrix, and before obtaining the mapping image of the projection picture taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively, the mapping image also includes:

[0112] A plurality of left image pairs are constructed based on the projection picture taken by the channel camera corresponding to the left channel and the projection picture taken by the viewpoint camera after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; a left image pair includes a first left image and a second left image; the first left image is the projection picture taken by the channel camera corresponding to the left channel after the position reference picture is projected onto the screen through the channel; the second left image is the projection picture taken by the viewpoint camera after the position reference picture is projected onto the screen through the channel; the channels and position reference pictures corresponding to the first left image and the second left image in the same left image pair are the same.

[0113] All left image pairs are traversed, and the pixel coordinates of the four corner points of each Apriltag in the left Apriltag set corresponding to the first left image in each left image pair are stored in the left original coordinate point set.

[0114] The pixel coordinates of the four corner points of each Apriltag in the left Apriltag set corresponding to the second left image in each left image pair are stored in the left coordinate point set to be mapped; the left Apriltag set corresponding to the first left image in the i-th left image pair and the left Apriltag set corresponding to the second left image in the i-th left image pair are the intersection of the Apriltag set of the first left image in the i-th left image pair and the Apriltag set of the second left image in the i-th left image pair.

[0115] The radial basis function interpolation method is used to interpolate the left original coordinate point set and the left coordinate point set to be mapped to obtain the left mapping matrix.

[0116] In another exemplary embodiment of the present application, after each position reference picture is projected onto the screen through the middle channel and the right channel respectively, the projection picture taken by the channel camera corresponding to the right channel is mapped according to the right mapping matrix, and before obtaining the mapping image of the projection picture taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively, the mapping image also includes:

[0117] A plurality of right image pairs are constructed based on the projection pictures taken by the channel camera corresponding to the right channel and the projection pictures taken by the viewpoint camera after the position reference picture is projected onto the screen through the middle channel and the right channel respectively; a right image pair includes a first right image and a second right image; the first right image is the projection picture taken by the channel camera corresponding to the right channel after the position reference picture is projected onto the screen through the channel; the second right image is the projection picture taken by the viewpoint camera after the position reference picture is projected onto the screen through the channel; the channels and position reference pictures corresponding to the first right image and the second right image in the same right image pair are the same.

[0118] All right image pairs are traversed, and the pixel coordinates of the four corner points of each Apriltag in the right Apriltag set corresponding to the first right image in each right image pair are stored in the right original coordinate point set.

[0119] The pixel coordinates of the four corner points of each Apriltag in the right side Apriltag set corresponding to the second right side image in each right side image pair are stored in the right side coordinate point set to be mapped; the right side Apriltag set corresponding to the first right side image in the i-th right side image pair and the right side Apriltag set corresponding to the second right side image in the i-th right side image pair are the intersection of the Apriltag set of the first right side image in the i-th right side image pair and the Apriltag set of the second right side image in the i-th right side image pair.

[0120] The radial basis function interpolation method is used to interpolate the right original coordinate point set and the right coordinate point set to be mapped to obtain the right mapping matrix.

[0121] In another exemplary embodiment of the present application, after each position reference picture is projected onto the screen through the intermediate channel, the projection picture taken by the channel camera corresponding to the intermediate channel is mapped according to the intermediate mapping matrix, and before obtaining the mapping image of the projection picture taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel, the mapping image also includes:

[0122] An intermediate image pair is constructed based on the projection image taken by the channel camera corresponding to the intermediate channel after each position reference image is projected onto the screen through the intermediate channel, the first complementary image and the second complementary image; an intermediate image pair includes the first intermediate image and the second intermediate image; when the first intermediate image is the projection image taken by the channel camera corresponding to the intermediate channel after the first position reference image is projected onto the screen through the intermediate channel, the second intermediate image is the first complementary image; when the first intermediate image is the projection image taken by the channel camera corresponding to the intermediate channel after the second position reference image is projected onto the screen through the intermediate channel, the second intermediate image is the second complementary image.

[0123] All intermediate image pairs are traversed, and the pixel coordinates of the four corner points of each Apriltag in the intermediate Apriltag set corresponding to the first intermediate image in each intermediate image pair are stored in the intermediate original coordinate point set.

[0124] The pixel coordinates of the four corner points of each Apriltag in the intermediate Apriltag set corresponding to the second intermediate image in each intermediate image pair are stored in the intermediate coordinate point set to be mapped; the intermediate Apriltag set corresponding to the first intermediate image in the i-th intermediate image pair and the intermediate Apriltag set corresponding to the second intermediate image in the i-th intermediate image pair are the intersection of the Apriltag set of the first intermediate image in the i-th intermediate image pair and the Apriltag set of the second intermediate image in the i-th intermediate image pair.

[0125] The intermediate original coordinate point set and the intermediate coordinate point set to be mapped are interpolated using the radial basis function interpolation method to obtain an intermediate mapping matrix.

[0126] In another exemplary embodiment of the present application, a comprehensive mapping matrix is ​​used to map the projection images of the projection images taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel, the projection images of the projection images taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel, and the projection images of the projection images taken by the channel camera corresponding to the middle channel after each position reference picture is projected onto the screen through the middle channel, and the mapped images are interpolated and inversely mapped to obtain the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel, and the following is also included:

[0127] The composite image is obtained according to the mapping image of the projection image taken by the channel camera corresponding to the left channel after the first position reference image is projected onto the screen through the left channel, the mapping image of the projection image taken by the channel camera corresponding to the right channel after the first position reference image is projected onto the screen through the right channel, and the mapping image of the projection image taken by the channel camera corresponding to the middle channel after the first position reference image is projected onto the screen through the middle channel. Specifically, the three images are added together to obtain the composite image.

[0128] The coordinates of each boundary point of the comprehensive image are sequentially perspective transformed and modified to obtain modified coordinates of each boundary point of the comprehensive image.

[0129] The coordinates of each boundary point of the integrated image are stored in the integrated original coordinate point set.

[0130] The modified coordinates of each boundary point of the comprehensive image are stored in a comprehensive coordinate point set to be mapped.

[0131] The radial basis function interpolation method is used to interpolate the comprehensive original coordinate point set and the comprehensive coordinate point set to be mapped to obtain a comprehensive mapping matrix.

[0132] In another exemplary embodiment of the present application, a comprehensive mapping matrix is ​​used to map the projection images of the projection images taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel, the projection images of the projection images taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel, and the projection images of the projection images taken by the channel camera corresponding to the middle channel after each position reference picture is projected onto the screen through the middle channel, and the mapped images are interpolated and inversely mapped to obtain the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel, specifically including:

[0133] A comprehensive mapping matrix is ​​used to map the mapping images of the projection pictures taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel, so as to obtain the first mapping image of the projection picture taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel.

[0134] A comprehensive mapping matrix is ​​used to map the mapping images of the projection pictures taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel, so as to obtain the first mapping image of the projection picture taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel.

[0135] A comprehensive mapping matrix is ​​used to map the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference image is projected onto the screen through the middle channel, so as to obtain a first mapping image of the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference image is projected onto the screen through the middle channel.

[0136] After projecting each position reference image onto the screen through the left channel respectively, the first mapping image of the projection image taken by the channel camera corresponding to the left channel is interpolated and inversely mapped to obtain an inverse mapping relationship corresponding to the left channel.

[0137] After projecting the reference pictures at each position onto the screen through the right channel respectively, the first mapping image of the projection picture taken by the channel camera corresponding to the right channel is interpolated and inversely mapped to obtain the inverse mapping relationship corresponding to the right channel.

[0138] After each position reference image is projected onto the screen through the intermediate channel, the first mapping image of the mapping image of the projection image taken by the channel camera corresponding to the intermediate channel is interpolated and inversely mapped to obtain an inverse mapping relationship corresponding to the intermediate channel.

[0139] This application also provides a specific embodiment to introduce the above method in detail:

[0140] The position reference pictures used in this embodiment are composed of two, such as Figure 3 and Figure 4 As shown. The image size is 3840*2160 pixels. The Apriltag specification used in the image is 36h11. The size of each Apriltag is 120*120 pixels, and the spacing between adjacent Apriltags is 60 pixels. Figure 3 The coordinates are (0, 0) and placed in sequence. Figure 4 The coordinates are (60,60). Figure 3 and Figure 4 All Apriltags in the program have different ids, and each Apriltag can be distinguished during program recognition. This application first needs to configure the curved surface projection correction system. Take the three-channel real image imaging flight simulator vision system as an example ( Figure 2 ), which is also applicable to simulator vision with other numbers of channels. In the three-channel real-image flight simulator vision system, the left channel is A0, the middle channel is A1, and the right channel is A2. Each channel (A0, A1, A2) corresponds to a channel camera (CamA0, CamA1, CamA2). The channel camera is installed outside the cabin, and there is no object blocking it from the curtain. There is no precise requirement for the camera installation posture, as long as the camera's image can cover the curtain area corresponding to the channel. A viewpoint camera (CamS) is installed in the middle of the cabin, and its installation position is at the theoretical viewpoint of the vision system design. The camera is required to be installed facing straight ahead and as horizontal as possible, and the shooting angle needs to include all areas of the imaging curtain (which may be blocked by mechanical structures).

[0141] In this embodiment, the original image is a normal image to be displayed without pre-distortion. The projected image is an image after the original image is pre-distorted, and the image is projected by a projector. The camera captures the image by projecting the projected image onto a screen through a projector, and then capturing the image on the screen with a camera.

[0142] The specific steps of this embodiment are:

[0143] Step 1: The camera captures a reference picture of the location displayed by the storage projector.

[0144] In steps 1, 2, 3, 4, and 5, the visual system used in the test of this method enabled three projection channels A0, A1, and A2, and used channel cameras CamA0, CamA1, CamA2, and viewpoint camera CamS. In this step of this application, the projection and shooting of the position reference image are completed, and the image size of the camera shooting and storage is 1920*1080. Figures 5 to 9The resolution of each sub-image is 1920*1080. The specific sub-steps are as follows:

[0145] Step 1.1: After the first position reference image is projected onto the screen through channel A0, channel cameras CamA0, CamA1, CamA2 and viewpoint camera CamS are used to shoot. Figure 5 a, b, c and Figure 7 a in.

[0146] Step 1.2: After the second position reference image is projected onto the screen through channel A0, channel cameras CamA0, CamA1, CamA2 and viewpoint camera CamS are used to shoot. Figure 6 a, b, c and Figure 7 In d.

[0147] Step 1.3: After the first position reference image is projected onto the screen through channel A1, channel cameras CamA0, CamA1, CamA2 and viewpoint camera CamS are used to shoot. Figure 5 d, e, f and Figure 7 b in.

[0148] Step 1.4: After the second position reference image is projected onto the screen through channel A1, channel cameras CamA0, CamA1, CamA2 and viewpoint camera CamS are used to shoot. Figure 6 d, e, f and Figure 7 The e in.

[0149] Step 1.5: After the first position reference image is projected onto the screen through channel A2, channel cameras CamA0, CamA1, CamA2 and viewpoint camera CamS are used to shoot. Figure 5 g, h, i and Figure 7 c in.

[0150] Step 1.6: After the second position reference image is projected onto the screen through channel A1, channel cameras CamA0, CamA1, CamA2 and viewpoint camera CamS are used to shoot. Figure 6 g, h, i and Figure 7 f in.

[0151] Step 1.7: Transfer all the pictures taken by the camera back to the server computer for storage.

[0152] Step 2: Multi-view inference based on thin plate spline interpolation to complete the occluded image and process it.

[0153] This application requires Figure 7 a to Figure 7 The part of i in i that is blocked by the instrument panel and mechanical structure is inferred, that is, the purpose is to complete the image of the blocked part in the picture taken by the viewpoint camera CamS. The specific sub-steps are as follows:

[0154] Step 2.0: Create a new empty set: the original coordinate point set and the coordinate point set to be mapped.

[0155] Step 2.1: Identify using existing libraries Figure 5 The April tag of a in the Figure 5 The pixel position (pixel coordinate) relationship of the four corner points of Apriltag corresponding to each id (Apriltag number) in a is M5a; identification Figure 7 The April tag of a in the Figure 7 The pixel position relationship of the four corner points of Apriltag corresponding to each id in a in M7a; find the pixel positions with the same id in M5a and M7a, store the pixel positions with the same id in M5a into the original coordinate point set SetA0, and store the pixel positions with the same id in M7a into the coordinate point set to be mapped SetB0.

[0156] Step 2.2: Identification Figure 6 The April tag of a in the Figure 6 The pixel position relationship M6a of the four corner points of Apriltag corresponding to each id in a; identification Figure 7 The pixel position relationship of the four corner points of Apriltag corresponding to each id in d is M7d; find the pixel positions with the same id in M6a and M7d, store the pixel positions with the same id in M6a into the original coordinate point set SetA0, and store the pixel positions with the same id in M7d into the coordinate point set to be mapped SetB0.

[0157] Step 2.3: Identification Figure 5 The April tag of d in the Figure 5 The pixel position relationship M5d of the four corner points of Apriltag corresponding to each id in d; identify Figure 7 The pixel position relationship of the four corner points of Apriltag corresponding to each id in b in M7b; find the pixel positions with the same id in M5d and M7b, store the pixel positions with the same id in M5d into the original coordinate point set SetA0, and store the pixel positions with the same id in M7b into the coordinate point set to be mapped SetB0.

[0158] Step 2.4: Identification Figure 6 The April tag of d in gets Figure 6 The pixel position relationship M6d of the four corner points of Apriltag corresponding to each id in d; identify Figure 7 The pixel position relationship of the four corner points of Apriltag corresponding to each id in e is M7e; find the pixel positions with the same id in M6d and M7e, store the pixel positions with the same id in M6d into the original coordinate point set SetA0, and store the pixel positions with the same id in M7e into the coordinate point set to be mapped SetB0.

[0159] Step 2.5: Use the radial basis function interpolation method to interpolate the original coordinate point set SetA0 and the coordinate point set to be mapped SetB0 to obtain the left mapping matrix Map0. Its kernel function uses the thin plate spline interpolation function. The obtained left mapping matrix is ​​the coordinate of each pixel in the image after mapping. That is, the value of the jth row and ith column of the left mapping matrix is ​​(ui, vi), then the pixel coordinates of the pixel in the jth row and ith column of the image in the mapped image are (ui, vi).

[0160] Step 2.6: Identify using existing libraries Figure 5 The April tag of i in the result is Figure 5 The pixel position (pixel coordinate) relationship M5i of the four corner points of Apriltag corresponding to each id (Apriltag number) in i; identify Figure 7 The April tag of c in the Figure 7 The pixel position relationship of the four corner points of Apriltag corresponding to each id in c in M7c; find the pixel positions with the same id in M5i and M7c, store the pixel positions with the same id in M5i into the original coordinate point set SetA2, and store the pixel positions with the same id in M7c into the coordinate point set to be mapped SetB2.

[0161] Step 2.7: Identification Figure 6 The April tag of i in the result is Figure 6 The pixel position relationship M6i of the four corner points of Apriltag corresponding to each id in i; identify Figure 7 The pixel position relationship of the four corner points of Apriltag corresponding to each id in f in M7f; find the pixel positions with the same id in M6i and M7f, store the pixel positions with the same id in M6i into the original coordinate point set SetA2, and store the pixel positions with the same id in M7f into the coordinate point set to be mapped SetB2.

[0162] Step 2.8: Identification Figure 5 The April tag of f in gets Figure 5 The pixel position relationship M5f of the four corner points of Apriltag corresponding to each id in f; identify Figure 7The pixel position relationship of the four corner points of Apriltag corresponding to each id in b in M7b; find the pixel positions with the same id in M5f and M7b, store the pixel positions with the same id in M5f into the original coordinate point set SetA2, and store the pixel positions with the same id in M7b into the coordinate point set to be mapped SetB2.

[0163] Step 2.9: Identification Figure 6 The April tag of f in gets Figure 6 The pixel position relationship M6f of the four corner points of Apriltag corresponding to each id in f; identify Figure 7 The pixel position relationship of the four corner points of Apriltag corresponding to each id in e in M7e; find the pixel positions with the same id in M6f and M7e, store the pixel positions with the same id in M6f into the original coordinate point set SetA2, and store the pixel positions with the same id in M7e into the coordinate point set to be mapped SetB2.

[0164] Step 2.10: Use the radial basis function interpolation method to interpolate the original coordinate point set SetA2 and the coordinate point set to be mapped SetB2 to obtain the right mapping matrix Map2. Its kernel function uses the thin plate spline interpolation function. The obtained right mapping matrix is ​​the coordinate of each pixel in the image after mapping. That is, the value of the jth row and ith column of the right mapping matrix is ​​(ui, vi), then the pixel coordinates of the pixel in the jth row and ith column of the image in the mapped image are (ui, vi).

[0165] Step 2.11: Use the left mapping matrix Map0 to Figure 5 a and d in Figure 6 The a and d in are mapped respectively to obtain Figure 8 a and d in Fig. 9 a and d in. Use the right mapping matrix Map2 to Figure 5 i and f in Figure 6 The i and f in are mapped respectively to obtain Figure 8 c and f in Fig. 9 c and f in. Figure 8 a and Fig. 9 a in the figure is the unobstructed position reference image projected through channel A0 at the viewpoint camera CamS inferred by thin plate spline interpolation. Figure 8 c in 9 and c in 9 are the unobstructed position reference images projected through channel A2 at the viewpoint camera CamS inferred by thin plate spline interpolation. Figure 8 d in , f in 8 and Figure 7 Add the b in Figure 8 The b in is the first completed image mentioned above. Fig. 9 d in 9, f in 9 and Figure 7 Add the e in Fig. 9 The b in is the second completed image mentioned above. Figure 8 b in and b in 9 are the blocked parts of channel A1 under the perspective of the complementary viewpoint camera CamS.

[0166] Step 2.12: Identification Figure 5 The April tag of e in Figure 5 The pixel position relationship of the four corner points of Apriltag corresponding to each id in e is M5e; identification Figure 8 The pixel position relationship of the four corner points of Apriltag corresponding to each id in b in M8b; find the pixel positions with the same id in M5e and M8b, store the pixel positions with the same id in M5e into the original coordinate point set SetA1, and store the pixel positions with the same id in M8b into the coordinate point set to be mapped SetB1.

[0167] Step 2.13: Identification Figure 6 The April tag of e in Figure 6 The pixel position relationship of the four corner points of Apriltag corresponding to each id in e is M6e; identification Fig. 9 The pixel position relationship of the four corner points of Apriltag corresponding to each id in b in M9b; find the pixel positions with the same id in M6e and M9b, store the pixel positions with the same id in M6e into the original coordinate point set SetA1, and store the pixel positions with the same id in M9b into the coordinate point set to be mapped SetB1.

[0168] Step 2.14: The original coordinate point set and the coordinate point set to be mapped are interpolated using the radial basis function interpolation method to obtain the intermediate mapping matrix Map1. The kernel function uses the thin plate spline interpolation function.

[0169] Step 2.15: Use the intermediate mapping matrix Map1 to Figure 5 e and Figure 6 The e in the equations are mapped to obtain Figure 8 e and Fig. 9 The e in. Figure 8 e in and e in 9 are the images projected through channel A1 with the unobstructed viewing angle of the viewpoint camera CamS inferred by thin plate spline interpolation. Figure 8 a to f and Fig. 9 From a to f in the figure are images of the viewpoint camera inferred from the channel camera.

[0170] Step 2.16: Figure 8 Add a, c, and e in Fig.10 , that is, the position reference map of all channels is integrated to display the screen boundary. Fig.10 The projection screen boundary shown in , obtains a series of boundary point sets Pe.

[0171] Step 2.17: Calculate the perspective transformation matrix H based on the top left, top right, bottom left, and bottom right points in the selected screen boundary. The pixel coordinates are marked as (x1, y1), (x2, y2), (x3, y3), (x4, y4), corresponding to coordinates (0, 0), (1920, 0), (0, 1080), (1920, 1080). The pixel coordinates after mapping using the perspective transformation matrix H are marked as (u1, v1), (u2, v2), (u3, v3), (u4, v4).

[0172] The perspective transformation takes the form:

[0173]

[0174] Substituting (x1, y1), (x2, y2), (x3, y3), (x4, y4) and (u1, v1), (u2, v2), (u3, v3), (u4, v4) into the above formula, we get:

[0175] The perspective transformation matrix H can be solved.

[0176] Perform perspective transformation on Pe according to H and calculate the point set Pp after perspective transformation. The coordinates of Pe are marked as (x Pe ,y Pe ), the coordinates of point Pp are marked as (x Pp ,y Pp ), x′, y′, w′ in the formula are all intermediate variables and have no practical significance. The formula for calculating Pp from Pe according to H is: and

[0177] Step 2.18: Modify the points in Pp. For a point in Pp, if this point is at the upper boundary of the screen, then the y coordinate of this point is set to 0; if this point is at the lower boundary of the screen, then the y coordinate of this point is set to 1080; if this point is at the left boundary of the screen, then the x coordinate of this point is set to 0; if this point is at the right boundary of the screen, then the x coordinate of this point is set to 1920. The point set after the modified coordinates is recorded as Pc.

[0178] Step 2.19: Take the point set Pe as the original coordinate point set and Pc as the coordinate point set to be mapped, and use the radial basis function interpolation method to interpolate to obtain the comprehensive mapping matrix Map3. The kernel function uses the thin plate spline interpolation function.

[0179] Step 2.20: Use the comprehensive mapping matrix Map3 to Figure 8 a, c and e in Fig. 9 The a, c and e in are mapped respectively to obtain Figure 8 g, h and i in Fig. 9 The purpose of g, h and i in the figure is to eliminate the influence of spherical imaging distortion, camera imaging distortion and the like on the image, and to display the screen in a full manner on the image. In step 2 of this embodiment, by taking a position reference picture of the projection of the complete screen at other viewing angles, based on the position reference picture of the unobstructed part in the picture taken by the camera at the viewpoint, it is inferred that the projection information of the obstructed part is restored without the need for calibration points on the screen, does not rely on the position information between the camera and the projector, and can be corrected when the viewpoint camera is blocked by a mechanical structure, resulting in missing images.

[0180] Step 3: Interpolation inverse mapping and correction.

[0181] In step 2, the position reference map is projected onto the screen area of ​​each channel under the viewpoint, and then the image is processed by filling the screen. Figure 8 g, h and i in Fig. 9 The length and width of the image in g, h, and i) correspond to the horizontal field of view of the screen display. For example, the field of view of the flight simulator is designed to be 180° horizontally and 40° vertically (the upper left point is (0°, 0°)). Figure 8 g, h and i in Fig. 9 In g, h, and i, the (0,0) coordinate indicates that the point displayed on the screen is 0 degrees horizontally and 0 degrees vertically, the (960,540) coordinate indicates that the point displayed on the screen is 90 degrees horizontally and 20 degrees vertically, and the (1920,1080) coordinate indicates that the point displayed on the screen is 180 degrees horizontally and 40 degrees vertically.

[0182] The image corresponding to channel A0 is Figure 8 g and Fig. 9 In g, the image corresponding to channel A1 is Figure 8 h and Fig. 9 In h, the image corresponding to channel A2 is Figure 8 The i and Fig. 9 The i in , respectively, shows the first position reference picture and the second position reference picture. When generating the first position reference picture, the pixel position relationship MA of the four corner points of the April tag corresponding to each id in the first position reference picture is known. When generating the second position reference picture, the pixel position relationship MB of the four corner points corresponding to each April tag id in the second position reference picture is known.

[0183] Perform the following steps for the two images corresponding to each channel, starting with the A0 channel. Figure 8 g and Fig. 9 Take g in as an example:

[0184] Step 3.1: Identification Figure 8 The pixel position relationship Mag of the four corner points of Apriltag corresponding to each id (Apriltag number) is obtained by taking the corner point position of each Apriltag in g. A known method is used to fit a polynomial curve to the corner points of each row and column, and then calculate the intersection of the curves of each row and column. After recalculating the intersection, the pixel position relationship Mag' of the four corner points of Apriltag corresponding to the id is obtained.

[0185] The steps to get Mag' are:

[0186] (1) Identification Figure 8 g in which the pixel positions of the four corner points of each Apriltag.

[0187] (2) According to the first position reference image and the second position reference image, count the corner points of each row and each column.

[0188] (3) For each row of corner points, a smooth curve is fitted using the polynomial fitting method. The coordinates of the corner points in the jth row are denoted as (x j ,y j ), the polynomial fitting method is used to fit the smooth curve C corresponding to the j-th row corner point j The polynomial fitting method formula is: M = 3. Among them, w represents each parameter of the polynomial fitting, M represents the highest order of the polynomial term, k represents the order of a term in the polynomial, and w kj is the polynomial parameter to be solved, y j (x j ,w) represents the fitted polynomial function, namely C j .

[0189] (4) For each column of corner points, a smooth curve is fitted using the polynomial fitting method. The i-th column of corner points is denoted by (x i ,y i ), the polynomial fitting method is used to fit the smooth curve C corresponding to the corner point in the i-th column i The polynomial fitting method formula is: M = 3. Among them, w ki is the polynomial parameter to be solved, x i (y i ,w) represents the fitted polynomial function, namely C i .

[0190] (5) For each transverse curve C j , calculate the value of each longitudinal C iThe calculated intersection coordinates are (p ci ,p cj ).

[0191] (6) In Mag, the ID of the April tag to which a corner point belongs can be calculated based on the information in the jth row and the ith column. For example, in the first position reference picture illustrated in the present application, the ID of the April tag in the second row and the first column is 22. Then the upper right corner of this April tag can be expressed as (p c2 ,p c3 ), that is, in the fitted curve, it is on the horizontal curve of j = 3 and on the vertical curve of i = 2. According to this correspondence, the coordinates of the corner points in the mapping relationship Mag can be updated to the newly calculated intersection coordinates, and the updated mapping relationship is recorded as Mag'.

[0192] Step 3.2: Identification Fig. 9 The corner point position of each Apriltag in g is obtained to obtain the pixel position relationship Mbg of the four corner points of the Apriltag corresponding to each id (Apriltag number). A polynomial curve is fitted for the corner points of each row and column using a known method. Then the intersection of the curves of each row and column is calculated. After recalculating the intersection, the pixel position relationship Mbg' of the four corner points of the Apriltag corresponding to the id is obtained.

[0193] Step 3.3: Assume that the image size of the original image (i.e., the first position reference image and the second position reference image) is m*n. Take Mag' and Mbg' as the original coordinate point set, take MA and MB as the coordinate point set to be mapped, and use the SmoothBivariateSpline method to interpolate to obtain the mapping matrix Map4. The size of the mapping matrix Map4 is 2m*2n. The element with coordinates (i, j) in the matrix represents the coordinates of the point with coordinates (i / 2, j / 2) on the original image corresponding to the coordinates on the projection image.

[0194] Step 3.4: Let the original image coordinate point set be W, and map W through the mapping matrix Map4 to obtain the coordinate point set on the image to be displayed as W′. There exists a mapping relationship f, namely the above-mentioned Map4, such that f(x p ,y p )=(x′ p ,y′ p ). However, due to (x′ p ,y′ p) is a floating point number. There is floating point truncation on the image, which results in some points not having a mapping relationship and some points having multiple mapping relationships. Therefore, this mapping relationship is injective but not surjective. Moiré or aliasing is caused on the image to be mapped, so inverse mapping is required. That is, according to the mapping relationship f, calculate the new inverse mapping relationship f -1 , so that f -1 (x′ p ,y′ p )=(x p ,y p ). Given (x p ,y p ) corresponds to the coordinate (x′) on the projection image. p ,y′ p ), according to (x′ p ,y′ p ), (x′ p+1 ,y′ p ), (x′ p ,y′ p+1 ), (x′ p+1 ,y′ p+1 ) and (x q+1 ,y q+1 ) can be inferred by bilinear interpolation. q+1 ,y q+1 ) corresponds to (x q ' +1 ,y q ' +1 ) pixel position, the inverse mapping diagram is as follows Fig.11 Finally, this step can obtain the coordinates of each pixel on the projected image corresponding to the original image pixel, completing the inverse mapping. This mapping relationship f -1 This is a well-known step.

[0195] According to the above steps 3.1 to 3.4, the A1 channel and the A2 channel each calculate the inverse mapping relationship M_inv of their corresponding channels. The three channels use their corresponding M_inv to correct the image as Fig.12 The pre-distortion diagram formed by pre-distortion is shown in Fig.13 As shown in a, b and c in the figure, project it onto the screen to complete the calibration. Fig.14 As shown, Fig.14 The a in the Fig.13 a in the image is projected onto the screen through the A0 channel and part of Fig.13 The actual picture of b being projected onto the screen through channel A1; Fig.14 The b in the Fig.13 The a in the image is projected onto the screen through the A0 channel and Fig.13 b is a real shot of the overlapping area of ​​the two channels projected onto the screen through channel A1; Fig.14 The c in the Fig.13 b is projected onto the screen through channel A1 and Fig.13 c is a real shot of the overlapping area of ​​the two channels projected onto the screen through channel A2; Fig.14 The d in the figure is the part Fig.13 b is projected onto the screen through channel A1 and Fig.13 Actual picture of c being projected onto the screen through the A2 channel.

[0196] This application has the following significant advantages:

[0197] 1. This application adopts a multi-view inference method to solve the problem that the viewpoint camera's field of view is blocked by the dashboard, and can complete the correction task of the visual system when it is blocked.

[0198] 2. This application does not require the use of calibration points on the screen. It is suitable for screens made of special materials that cannot be marked on them. At the same time, if the calibration points on the screen are not manually marked, the preparation time before calibration can be greatly reduced.

[0199] 3. This application significantly reduces the calibration time. It takes only about ten minutes to complete the calibration of three channels, which is much less than the several hours taken by the previous method.

[0200] 4. Solve the problem of occlusion at the viewpoint without manual calibration in the prior art without using the curtain calibration points.

[0201] 5. In step three of the present application, interpolation operation is performed to complete rough correction, which greatly improves the correction speed and robustness.

[0202] Based on the same inventive concept, the embodiment of the present application also provides a multi-channel curved surface projection stitching correction device for implementing the multi-channel curved surface projection stitching correction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multi-channel curved surface projection stitching correction device embodiments provided below can refer to the limitations of the multi-channel curved surface projection stitching correction method above, and will not be repeated here.

[0203] In an exemplary embodiment, a multi-channel curved surface projection stitching correction device is provided, comprising:

[0204] The acquisition module is used to project the reference images of each position onto the screen through each channel, and then the viewpoint camera and each channel camera take the projection image; the channels include the left channel, the middle channel and the right channel, and one channel corresponds to one channel camera.

[0205] The first mapping module is used to map the projection pictures taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively according to the left mapping matrix, so as to obtain a mapping image of the projection pictures taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; the left mapping matrix is ​​obtained based on the projection pictures taken by the channel camera corresponding to the left channel and the projection pictures taken by the viewpoint camera after each position reference picture is projected onto the screen through the left channel and the middle channel respectively.

[0206] The second mapping module is used to map the projection pictures taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively according to the right mapping matrix, so as to obtain a mapping image of the projection pictures taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively; the right mapping matrix is ​​obtained based on the projection pictures taken by the channel camera corresponding to the right channel and the projection pictures taken by the viewpoint camera after each position reference picture is projected onto the screen through the middle channel and the right channel respectively.

[0207] The first completion module is used to obtain a first completion image based on a projection image taken by a rear view camera after the first position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the first position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the first position reference image is projected onto the screen through the middle channel.

[0208] The second completion module is used to obtain a second completion image based on the projection image taken by the rear view camera after the second position reference image is projected onto the screen through the middle channel, the mapping image of the projection image taken by the channel camera corresponding to the left channel after the second position reference image is projected onto the screen through the middle channel, and the mapping image of the projection image taken by the channel camera corresponding to the right channel after the second position reference image is projected onto the screen through the middle channel.

[0209] The third mapping module is used to map the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel according to the intermediate mapping matrix, and obtain a mapping image of the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel; the intermediate mapping matrix is ​​obtained based on the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel, the first complement image and the second complement image.

[0210] The inverse mapping module is used to use a comprehensive mapping matrix to map the mapping images of the projection images taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference picture is projected onto the screen through the middle channel, and perform interpolation inverse mapping operations on the mapped images to obtain the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel; the comprehensive mapping matrix is ​​obtained according to the mapping images of the projection images taken by the channel camera corresponding to the left channel after the first position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after the first position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after the first position reference picture is projected onto the screen through the middle channel.

[0211] The correction module is used to pre-distort the image to be corrected by using the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel and the inverse mapping relationship corresponding to the middle channel, and project the pre-distorted image to complete the correction of the image to be corrected.

[0212] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above-mentioned method embodiments are implemented when the processor executes the computer program.

[0213] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0214] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0215] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0216] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0217] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-channel curved surface projection correction method, characterized in that: The multi-channel curved surface projection correction method comprises: After projecting the reference images of each position onto the screen through each channel, the viewpoint camera and each channel camera take projection images; the channels include the left channel, the middle channel and the right channel, and one channel corresponds to one channel camera; After each position reference picture is projected onto the screen through the left channel and the middle channel respectively, the projection picture taken by the channel camera corresponding to the left channel is mapped according to the left mapping matrix, so as to obtain a mapping image of the projection picture taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; the left mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the left channel and the projection picture taken by the viewpoint camera after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; After each position reference picture is projected onto the screen through the middle channel and the right channel, the projection picture taken by the channel camera corresponding to the right channel is mapped according to the right mapping matrix, so as to obtain a mapping image of the projection picture taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively; the right mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the right channel and the projection picture taken by the viewpoint camera after each position reference picture is projected onto the screen through the middle channel and the right channel respectively; A first completed image is obtained according to a projection image taken by a rear viewpoint camera after the first position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the first position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the first position reference image is projected onto the screen through the middle channel; A second completed image is obtained according to a projection image taken by a rear viewpoint camera after the second position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the second position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the second position reference image is projected onto the screen through the middle channel; After each position reference picture is projected onto the screen through the intermediate channel, the projection picture taken by the channel camera corresponding to the intermediate channel is mapped according to the intermediate mapping matrix, so as to obtain a mapping image of the projection picture taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel; the intermediate mapping matrix is ​​obtained according to the projection picture taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel, the first complement image and the second complement image; A comprehensive mapping matrix is ​​used to map the mapping images of the projection images taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference picture is projected onto the screen through the middle channel, and an interpolation inverse mapping operation is performed on the mapped images to obtain an inverse mapping relationship corresponding to the left channel, an inverse mapping relationship corresponding to the right channel, and an inverse mapping relationship corresponding to the middle channel; the comprehensive mapping matrix is ​​obtained according to the mapping images of the projection images taken by the channel camera corresponding to the left channel after the first position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after the first position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after the first position reference picture is projected onto the screen through the middle channel; The image to be corrected is pre-distorted by using the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel, and the pre-distorted image is projected to complete the correction of the image to be corrected.

2. The multi-channel curved surface projection correction method according to claim 1, characterized in that: According to the left mapping matrix, after each position reference picture is projected onto the screen through the left channel and the middle channel, the projection picture taken by the channel camera corresponding to the left channel is mapped to obtain the mapping image of the projection picture taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively, and the mapping image also includes: A plurality of left image pairs are constructed according to the projection pictures taken by the channel camera corresponding to the left channel and the projection pictures taken by the viewpoint camera after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; a left image pair includes a first left image and a second left image; the first left image is the projection picture taken by the channel camera corresponding to the left channel after the position reference picture is projected onto the screen through the channel; the second left image is the projection picture taken by the viewpoint camera after the position reference picture is projected onto the screen through the channel; the channels and position reference pictures corresponding to the first left image and the second left image in the same left image pair are the same; Traverse all left image pairs, and store the pixel coordinates of the four corner points of each Apriltag in the left Apriltag set corresponding to the first left image in each left image pair into the left original coordinate point set; The pixel coordinates of the four corner points of each Apriltag in the left Apriltag set corresponding to the second left image in each left image pair are stored in the left coordinate point set to be mapped; the left Apriltag set corresponding to the first left image in the i-th left image pair and the left Apriltag set corresponding to the second left image in the i-th left image pair are the intersection of the Apriltag set of the first left image in the i-th left image pair and the Apriltag set of the second left image in the i-th left image pair; The radial basis function interpolation method is used to interpolate the left original coordinate point set and the left coordinate point set to be mapped to obtain the left mapping matrix.

3. The multi-channel curved surface projection correction method according to claim 1, characterized in that: After each position reference picture is projected onto the screen through the middle channel and the right channel respectively, the projection picture taken by the channel camera corresponding to the right channel is mapped according to the right mapping matrix to obtain the mapping image of the projection picture taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively. The mapping image also includes: A plurality of right image pairs are constructed according to the projection pictures taken by the channel camera corresponding to the right channel and the projection pictures taken by the viewpoint camera after the position reference picture is projected onto the screen through the middle channel and the right channel respectively; a right image pair includes a first right image and a second right image; the first right image is the projection picture taken by the channel camera corresponding to the right channel after the position reference picture is projected onto the screen through the channel; the second right image is the projection picture taken by the viewpoint camera after the position reference picture is projected onto the screen through the channel; the channels and position reference pictures corresponding to the first right image and the second right image in the same right image pair are the same; Traverse all right image pairs, and store the pixel coordinates of the four corner points of each Apriltag in the right Apriltag set corresponding to the first right image in each right image pair into the right original coordinate point set; The pixel coordinates of the four corner points of each Apriltag in the right Apriltag set corresponding to the second right image in each right image pair are stored in the right coordinate point set to be mapped; the right Apriltag set corresponding to the first right image in the i-th right image pair and the right Apriltag set corresponding to the second right image in the i-th right image pair are the intersection of the Apriltag set of the first right image in the i-th right image pair and the Apriltag set of the second right image in the i-th right image pair; The radial basis function interpolation method is used to interpolate the right original coordinate point set and the right coordinate point set to be mapped to obtain the right mapping matrix.

4. The multi-channel curved surface projection correction method according to claim 1, characterized in that: After each position reference picture is projected onto the screen through the intermediate channel, the projection picture taken by the channel camera corresponding to the intermediate channel is mapped according to the intermediate mapping matrix to obtain the mapping image of the projection picture taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel. The method also includes: An intermediate image pair is constructed according to the projection image taken by the channel camera corresponding to the intermediate channel after each position reference image is projected onto the screen through the intermediate channel, the first complementary image and the second complementary image; an intermediate image pair includes the first intermediate image and the second intermediate image; when the first intermediate image is the projection image taken by the channel camera corresponding to the intermediate channel after the first position reference image is projected onto the screen through the intermediate channel, the second intermediate image is the first complementary image; when the first intermediate image is the projection image taken by the channel camera corresponding to the intermediate channel after the second position reference image is projected onto the screen through the intermediate channel, the second intermediate image is the second complementary image; Traverse all intermediate image pairs, and store the pixel coordinates of the four corner points of each Apriltag in the intermediate Apriltag set corresponding to the first intermediate image in each intermediate image pair into the intermediate original coordinate point set; The pixel coordinates of the four corner points of each Apriltag in the intermediate Apriltag set corresponding to the second intermediate image in each intermediate image pair are stored in the intermediate coordinate point set to be mapped; the intermediate Apriltag set corresponding to the first intermediate image in the i-th intermediate image pair and the intermediate Apriltag set corresponding to the second intermediate image in the i-th intermediate image pair are the intersection of the Apriltag set of the first intermediate image in the i-th intermediate image pair and the Apriltag set of the second intermediate image in the i-th intermediate image pair; The intermediate original coordinate point set and the intermediate coordinate point set to be mapped are interpolated using the radial basis function interpolation method to obtain an intermediate mapping matrix.

5. The multi-channel curved surface projection correction method according to claim 1, characterized in that: Before mapping the mapping images of the projection images taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference picture is projected onto the screen through the middle channel using the comprehensive mapping matrix, it also includes: A comprehensive image is obtained according to a mapping image of a projection image taken by a channel camera corresponding to the left channel after the first position reference image is projected onto the screen through the left channel, a mapping image of a projection image taken by a channel camera corresponding to the right channel after the first position reference image is projected onto the screen through the right channel, and a mapping image of a projection image taken by a channel camera corresponding to the middle channel after the first position reference image is projected onto the screen through the middle channel; Performing perspective transformation and modification on the coordinates of each boundary point of the comprehensive image in sequence to obtain modified coordinates of each boundary point of the comprehensive image; Storing the coordinates of each boundary point of the comprehensive image into a comprehensive original coordinate point set; Storing the modified coordinates of each boundary point of the comprehensive image into a comprehensive coordinate point set to be mapped; The radial basis function interpolation method is used to interpolate the comprehensive original coordinate point set and the comprehensive coordinate point set to be mapped to obtain a comprehensive mapping matrix.

6. The multi-channel curved surface projection correction method according to claim 5, characterized in that: A comprehensive mapping matrix is ​​used to map the projection images of the projection images taken by the channel camera corresponding to the left channel after each position reference image is projected onto the screen through the left channel, the projection images of the projection images taken by the channel camera corresponding to the right channel after each position reference image is projected onto the screen through the right channel, and the projection images of the projection images taken by the channel camera corresponding to the middle channel after each position reference image is projected onto the screen through the middle channel, and an interpolation inverse mapping operation is performed on the mapped images to obtain an inverse mapping relationship corresponding to the left channel, an inverse mapping relationship corresponding to the right channel, and an inverse mapping relationship corresponding to the middle channel, specifically including: Use a comprehensive mapping matrix to map the mapping images of the projection images taken by the channel camera corresponding to the left channel after each position reference image is projected onto the screen through the left channel, to obtain a first mapping image of the projection image taken by the channel camera corresponding to the left channel after each position reference image is projected onto the screen through the left channel; Use a comprehensive mapping matrix to map the mapping images of the projection images taken by the channel camera corresponding to the right channel after each position reference image is projected onto the screen through the right channel, to obtain a first mapping image of the projection images taken by the channel camera corresponding to the right channel after each position reference image is projected onto the screen through the right channel; After each position reference picture is projected onto the screen through the middle channel, the mapping image of the projection picture taken by the channel camera corresponding to the middle channel is mapped using the comprehensive mapping matrix to obtain a first mapping image of the mapping image of the projection picture taken by the channel camera corresponding to the middle channel after each position reference picture is projected onto the screen through the middle channel; After projecting the reference pictures at each position onto the screen through the left channel respectively, interpolating and inverse mapping the first mapping image of the projection picture taken by the channel camera corresponding to the left channel to obtain an inverse mapping relationship corresponding to the left channel; After the reference pictures at each position are projected onto the screen through the right channel respectively, the first mapping image of the projection picture taken by the channel camera corresponding to the right channel is interpolated and inversely mapped to obtain an inverse mapping relationship corresponding to the right channel; After each position reference image is projected onto the screen through the intermediate channel, the first mapping image of the mapping image of the projection image taken by the channel camera corresponding to the intermediate channel is interpolated and inversely mapped to obtain an inverse mapping relationship corresponding to the intermediate channel.

7. A multi-channel curved surface projection correction device, characterized in that: The multi-channel curved surface projection correction device comprises: The acquisition module is used to project the reference images of each position onto the screen through each channel, and then the viewpoint camera and each channel camera take the projection images; the channels include the left channel, the middle channel and the right channel, and one channel corresponds to one channel camera; A first mapping module is used to map the projection pictures taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively according to the left mapping matrix, so as to obtain a mapping image of the projection pictures taken by the channel camera corresponding to the left channel after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; the left mapping matrix is ​​obtained according to the projection pictures taken by the channel camera corresponding to the left channel and the projection pictures taken by the viewpoint camera after each position reference picture is projected onto the screen through the left channel and the middle channel respectively; A second mapping module is used to map the projection pictures taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively according to the right mapping matrix, so as to obtain a mapping image of the projection pictures taken by the channel camera corresponding to the right channel after each position reference picture is projected onto the screen through the middle channel and the right channel respectively; the right mapping matrix is ​​obtained according to the projection pictures taken by the channel camera corresponding to the right channel and the projection pictures taken by the viewpoint camera after each position reference picture is projected onto the screen through the middle channel and the right channel respectively; A first completion module, for obtaining a first completion image according to a projection image taken by a rear viewpoint camera after the first position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the first position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the first position reference image is projected onto the screen through the middle channel; A second completion module is used to obtain a second completion image according to a projection image taken by a rear viewpoint camera after the second position reference image is projected onto the screen through the middle channel, a mapping image of a projection image taken by a channel camera corresponding to a left channel after the second position reference image is projected onto the screen through the middle channel, and a mapping image of a projection image taken by a channel camera corresponding to a right channel after the second position reference image is projected onto the screen through the middle channel; A third mapping module is used to map the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel according to the intermediate mapping matrix, so as to obtain a mapping image of the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel; the intermediate mapping matrix is ​​obtained according to the projection pictures taken by the channel camera corresponding to the intermediate channel after each position reference picture is projected onto the screen through the intermediate channel, the first complement image and the second complement image; An inverse mapping module is used to use a comprehensive mapping matrix to map the mapping images of the projection images taken by the channel camera corresponding to the left channel after each position reference image is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after each position reference image is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after each position reference image is projected onto the screen through the middle channel, and perform interpolation inverse mapping operations on the mapped images to obtain the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel, and the inverse mapping relationship corresponding to the middle channel; the comprehensive mapping matrix is ​​obtained according to the mapping images of the projection images taken by the channel camera corresponding to the left channel after the first position reference image is projected onto the screen through the left channel, the mapping images of the projection images taken by the channel camera corresponding to the right channel after the first position reference image is projected onto the screen through the right channel, and the mapping images of the projection images taken by the channel camera corresponding to the middle channel after the first position reference image is projected onto the screen through the middle channel; The correction module is used to pre-distort the image to be corrected by using the inverse mapping relationship corresponding to the left channel, the inverse mapping relationship corresponding to the right channel and the inverse mapping relationship corresponding to the middle channel, and project the pre-distorted image to complete the correction of the image to be corrected.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-channel curved surface projection correction method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-channel curved surface projection correction method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-channel curved surface projection correction method described in any one of claims 1 to 6 is implemented.

Images