An automatic curved surface projection correction splicing method

Abstract

The application discloses an automatic curved surface projection correction splicing method, first, a network camera driving module is run, file mapping technology is used to transmit image information with a main program, then a curtain calibration point is recognized according to a specific color. After that, through an iterative method, in each round of iteration, a core algorithm is used to update distortion parameters according to a projection calibration diagram collected by a camera until a termination condition is reached and then the method is stopped. Finally, according to the distortion parameters obtained through iteration, a final corrected image is obtained. After distortion correction, taking an intermediate channel image as a reference, an iterative splicing correction method is used in left and right channels to complete splicing correction. Finally, in order to eliminate the bright band caused by splicing, color transition is performed on the image. The application does not depend on the accurate position relationship between the camera and the projector, the model can automatically iterate to complete distortion correction and splicing work, and manual participation is not needed, so that the problems of low precision and long time consumption in manual correction are solved, and the application has good correction, splicing and color transition effects.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of flight simulation technology, and particularly relates to an automatic curved surface projection correction splicing method. BACKGROUND

[0002] In most current flight simulation devices, projection is used for view imaging. The projection is real image imaging on an actual screen, and is virtual image imaging after multiple lens optical processing. In order to achieve a realistic effect during simulation, a large field of view and multiple channels are used for projection. In most devices, a manual correction method is used for pre-distortion correction of the projected picture, and the correction result is greatly dependent on human subjective feeling, and the precision is low. At the same time, since it is manual operation, a long time is consumed for each correction.

[0003] During the running of a flight simulation platform, in order to simulate weightlessness and overweight feeling, the whole simulation cabin is often moved, and the projection device which is fixed and installed may be deviated due to shaking, so that ghosting is generated in the projection splicing area of adjacent channels. Therefore, an efficient, fully automatic and high-precision correction splicing tool is essential. SUMMARY

[0004] In view of the above problems, the application provides an automatic curved surface projection correction splicing method. In order to solve the problems of low precision and long time consumption in manual correction, an iterative correction core method is used, and an iterative correction method based on a spherical model and an iterative correction method based on a curve intersection model are used to automatically complete correction, thereby greatly reducing the time and labor cost required for correction on the basis of ensuring precision.

[0005] The automatic curved surface projection correction splicing method comprises the following steps.

[0006] Step 1: under the condition that the projection devices of three channels are closed and the ultraviolet lamp is opened, the screen under each channel is photographed by the network cameras of the three channels respectively, and the pictures containing the screen calibration points are saved.

[0007] Step 2: the screen calibration points are processed to obtain clear screen calibration points.

[0008] Step 3: the screen calibration points are selected in order.

[0009] Step 4: Turn on the projector, and first correct the distortion of the left and right channels, and then correct the distortion of the middle channel. The distortion correction is performed by using the iterative correction method based on the intersection model of curves. The method is as follows: a region of M*N calibration points on the curtain is used to fit M+N curves, and a total of (M+N)*4 parameters are obtained. Then, the parameters are interpolated to deduce a plurality of curves equal to the number of pixels of the projection picture. The position of each pixel on the projection picture P after transformation is determined by the intersection of a horizontal and a vertical conic curve.

[0010] Step 5: After the distortion correction of all channels is completed, each channel uses the curve corner xy coordinate data obtained in step 4 to complete the mapping of the chessboard and is projected on the curtain. The Apriltag is used to generate a calibration map used for splicing the projection images of each channel. The middle channel further projects the projection calibration map, and the left and right channels project the base color map. The projection of the calibration map is identified by the left and right channels respectively, and the results are comprehensively calculated to obtain all the calibration anchor points of the splicing belt. Then, the left and right channels perform iterative splicing correction on the full black picture projected by the middle channel.

[0011] Step 6: The color transition method based on trigonometric functions is used to perform color transition processing on the splicing belt.

[0012] Step 7: The parameters calculated in the distortion iterative correction and iterative splicing correction are reversely mapped and the interpolation is used to eliminate the sawtooth.

[0013] The advantages of the present application are as follows:

[0014] 1. The automatic curved surface projection correction splicing method adopts a modular design, and the image acquisition part and the running part are independent, which is convenient for subsequent maintenance and upgrade.

[0015] 2. The automatic curved surface projection correction splicing method adopts the core method of iterative distortion correction, and uses the iterative correction method based on the intersection model of curves to automatically complete the correction without manual operation.

[0016] 3. The automatic curved surface projection correction splicing method reduces the time and steps required for correction, and the installation positions of the camera and the projector do not need to be strictly limited.

[0017] 3. The automatic curved surface projection correction splicing method adopts the anchor point splicing correction method based on iteration to realize automatic splicing of adjacent channels. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The automatic curved surface projection correction splicing method is a whole flow chart.

[0019] Figure 2 Flow chart for network camera image acquisition and image information communication design;

[0020] Figure 3 Schematic diagram of network camera image read-write mode;

[0021] Figure 4 Schematic diagram for generating a curve for distortion iteration correction;

[0022] Figure 5 Projection diagram of distortion correction result;

[0023] Figure 6 Effect diagram after linear interpolation of the curve in the distortion correction process;

[0024] Figure 7 Image after distortion correction;

[0025] Figure 8 Splicing band effect diagram when projecting adjacent channels;

[0026] Figure 9 AprilTag anchor point diagram actual photograph;

[0027] Figure 10 Original diagram of A and B two calibration diagrams and projection effect diagram of A and B seen from the camera perspective;

[0028] Figure 11 Effect of superimposing the black and white areas of the double-channel checkerboard after completing the splicing correction;

[0029] Figure 12 Splicing band effect diagram after color transition;

[0030] Figure 13 Color transition function based on trigonometric functions used in the present application. DETAILED DESCRIPTION

[0031] The present application will be further described in detail below in conjunction with the accompanying drawings.

[0032] The automatic curved surface projection correction splicing method of the present application, as shown in Figure 1 , the specific steps are as follows:

[0033] Step 1: Image acquisition and image information communication design.

[0034] Due to the fact that the image acquisition device is too far away from the host computer used for image processing during actual equipment installation and debugging, it is not very practical to use a USB camera to transmit to the host computer. Therefore, a network camera module is used for image acquisition in the present application. As shown in Figure 2As shown, first, the network camera module calls the camera SDK, initializes the camera, logs in, and starts real-time preview, sets the required resolution and frame rate. Then the played image is captured and transmitted through TCP. The network camera module runs the program as 32-bit, compiled and packaged into an executable file by Visual Studio, and called by the subprocess library of the main program python environment. The network camera module obtains a frame of image package through the interface H264_PLAY_CatchPicBuf, and shares memory through filemapping technology. In the network camera module running program, a TESTSM area with a size of 1920*1080*3 is opened; in the subsequent loop, each time the image is obtained, it is written into the memory, such as Figure 3 As shown in the main program, the mmap file mapping support is used to read the data package and convert it into a data format that can be processed by the image, and the specific method is as follows:

[0035] data=mmap.mmap(0,bufSize,tagname=FILENAME,access=mmap.ACCESS_READ)

[0036] dataRead=data.read(bufSize)

[0037] d=Image.open(io.BytesIO(dataRead))

[0038] img=cv2.cvtColor(np.asarray(d),cv2.COLOR_RGB2BGR)

[0039] Through the above method, the three channel projectors are closed and opened under the condition of ultraviolet lamp, and the three channel network cameras are used to shoot the curtains under the respective channels, and the pictures containing the curtain calibration points are saved.

[0040] Step 2: Identification and detection of curtain calibration points

[0041] From the picture containing the curtain calibration points read from the camera, it can be seen that the background color is not uniform and similar to the calibration point color, there are many noise points and other light spots, so that the curtain calibration point is not clear, and the effect of using traditional HSV color recognition, thresholding and other means is poor. Therefore, in order to obtain clear curtain calibration points for subsequent processing, the image difference method is used to process the curtain calibration points. Since the curtain calibration points are similar to large particle size pepper and salt noise, median filtering is used for processing.

[0042] After getting the clear curtain calibration points, the curtain calibration points are selected in order by manual calibration method. The following is the specific operation method.

[0043] After the camera reads the filtered curtain calibration point pictures saved by the camera, the "corners" window is displayed. The local picture area containing a single calibration point is selected manually by controlling the mouse, and the position of the calibration point is automatically detected according to the selected local area and circled with a red circle. The selection order is to select from the first available calibration point in the top left corner by row. After all the curtain calibration points are selected, the center coordinates of all the calibrations are calculated, that is, the average of all the calibration point x coordinates and the average of all the calibration point y coordinates, and the data is saved in the required format, including saving all the calibration points in row format and column format, and saving the calculated center point of all the calibration points.

[0044] Step 3: Iterative correction method based on curve intersection model for distortion iterative correction

[0045] After the manual selection of the curtain calibration points in the three channels is completed, the projector is turned on; due to the manual selection of the calibration points in step 2 and the turning on of the projector, there is a time gap between the front and back of each channel, so each channel is first automatically synchronized, and then each channel is subjected to distortion iterative correction; among them, the left channel and the right channel are first subjected to distortion iterative correction, and the middle channel waits for the other two channels to correct before performing distortion correction.

[0046] In the device for imaging a virtual image, since the curtain to be imaged is not a standard spherical curtain, the iterative correction method based on the spherical model cannot be used on this type of curtain. For the imaging correction of the non-standard spherical curved curtain, the iterative correction method based on the curve intersection model is used for distortion iterative correction, as follows:

[0047] On the curved curtain, the points with the same longitude and the same latitude satisfy the conic section equation, so a series of points on the projection picture P on the curtain satisfy the elliptic equation. At the same time, the size of the original picture P0 is 1080*1920, and the display area of the picture on the curtain is the area represented by the M rows and N columns of curtain calibration points on the curtain, and each curtain calibration point is on a latitude line and a longitude line. Similarly, each pixel of the original picture P0 corresponds to a point on the curved curtain, which is on a latitude line and a longitude line.

[0048] But it is not realistic to fit 1920*1080 pixels, so the present application adopts the region of M*N calibration points on the curtain to fit M+N curves, a total of (M+N)*4 parameters, and then performs interpolation operation on the parameters to deduce 1920+1080 curves. The position of each pixel on the projection picture P after transformation (the position of the pixel on the original picture P0 mapped to P) can be determined by the intersection of a horizontal and a vertical conic curve.

[0049] When using the above iterative correction method based on the curve intersection model for correction, there are two stages of fitting horizontal elliptic curves and fitting vertical elliptic curves. The two stages have the same execution process, and both use the iterative correction method.

[0050] As shown in Figure 4 , the curves obtained after iterative calculation of the 8 rows and 16 columns of curtain calibration points displayed on the projection picture P, Figure 5 the effect on the curved virtual image curtain after projection of the projection picture P. As can be seen from the figure, all the curtain calibration points are on the corresponding projected curves, and the calibration points coincide with the intersection points of the curves.

[0051] As shown in Figure 6 , the effect of linear interpolation of the curves, a total of 1080+1920 curves displayed on the projection picture P. As can be seen from the figure, the 1080 horizontal curves and the 1920 vertical curves all have intersection points, a total of 1080*1920 intersection points, corresponding to each pixel in the original picture. By matching these intersection points one by one with the pixels on the standard checkerboard, the distortion-corrected Figure 7 . Here, the Boeing 737 flight simulation cockpit is used for experiments. When shooting the splicing and correction results, it can only be done in the flight cockpit, so it is unavoidable to be blocked by the cockpit porthole columns, but it does not affect the display and judgment of the correction and splicing effect.

[0052] Step 4: Iterative splicing based on anchor points

[0053] After all the channels are distortion-corrected, each channel uses its own 1920+1080 curve corner point xy coordinate data (a total of (1920*1080)*2 parameters) obtained in step 3 to complete the mapping of the checkerboard and simultaneously project on the curtain. When adjacent channels are projected, there is an overlapping part of the image, which becomes a splicing band, and the splicing band effect is shown in Figure 8 As can be seen from the degree of coincidence of the checkerboard, the two channels do not completely coincide, and there is still a deviation; the deviation seems small, but in actual application it will have a larger blur, so the splicing band area needs to be corrected.

[0054] The current curtain exists two splicing bands, which are the splicing bands of left channel and middle channel, and the splicing bands of middle channel and right channel. The anchor points used in splicing are generated by using Apriltag. Figure 9 The two-dimensional code shown in the middle, there are 7 Apriltags on the calibration map corresponding to a splicing band, the sub-pixel positions of the four corners of the Apriltag image are calculated by identifying the Apriltag, and the positions of the four corners of each AprilTag image are the positions of the splicing band anchor points. Considering that there are even number of AprilTags in the splicing area, a single calibration map cannot completely represent all the calibration anchor points, therefore two calibration maps, A map and B map, are used.

[0055] Fix the middle channel, project A map and B map on the curtain respectively. When projecting in the middle channel, the left and right channels also need to project the base color picture; the area near the splicing band is black, and the other areas are white. The original map of A map and B map and the projection effect of A map and B map seen by the camera view angle are shown in Figure 10 This method has the advantage of improving recognition accuracy. Therefore, the left channel and the right channel respectively identify the projection of A map and B map, and then the comprehensive calculation result is obtained to obtain all the calibration anchor points of the splicing band.

[0056] After obtaining all the calibration anchor points of the splicing band, a full black picture is projected by the middle channel, and the left and right channels are iteratively spliced and corrected. The splicing flow chart is shown in the figure. The specific splicing correction method is: the distance difference dx and dy between the splicing point and the anchor point corresponding thereto, unit: pixel, is iteratively modified according to the distance difference. When |dx|<0.3 and |dy|<0.3, the iteration is stopped. If the splicing band anchor points used are 8 rows and 3 columns, the splicing correction parameters (the horizontal and vertical coordinate difference between the splicing point and the anchor point corresponding thereto) before interpolation are 8 rows and 16 columns in the left channel image; the first to the thirteenth columns are all 0, and the thirteenth / 14th / 15th columns are the splicing correction parameters obtained by iterative correction. In order to ensure the continuity of the image, the 1536th to 1920th columns of the 13th / 14th / 15th / 16th columns of the 1080*1920 image are bilinearly interpolated. Figure 11 The effect of the double-channel checkerboard black and white area after superposition after completing the splicing correction. Set the unit checkerboard edge length in the projection picture P as l p , and the checkerboard edge length in the camera shooting picture as l c . Define the splicing error as the ratio of the non-coincidence area of the checkerboard after splicing to the area itself.

[0057]

[0058] wherein, checkerboard l c= 128. Then, according to the iteration strategy, the end condition is corrected, and theoretically, the error after splicing is less than 1.4%.

[0059] As can be seen from the figure, the splicing error on each grid is small, and the effect after color transition is as shown in Figure 12 , wherein the square area is the splicing band.

[0060] Step 5: Color transition processing of the splicing band

[0061] If the color transition is not performed on the splicing band area, the area will be obviously brighter than other areas. Moreover, the sum of the theoretical brightness of the splicing band should be greater than 100% to project the splicing band smoothly on the screen. Therefore, in the present application, a color transition method based on a trigonometric function is used to perform color transition processing on the splicing band. The transition function used in the present application is:

[0062]

[0063] wherein A = 1, B = 1 / 11, and C = 1 / 80, which are obtained according to the continuity of the brightness of the transition band after debugging; x is the normalized splicing band coordinate. The linear transition function is as shown in Figure 13 , wherein f t (x) represents the brightness when the left channel splicing band is projected, f t (x) represents the brightness when the right channel splicing band is projected, f t (x) + f t (1-x) represents the sum of the theoretical projection brightness.

[0064] Step 6: Visualization of the corrected splicing result

[0065] The correction parameters obtained by iteration are visualized. Since the parameters calculated in steps 4 and 5 (the position of each pixel of the original image P0 corresponding to the position on the projection image P after splicing is completed, and the brightness coefficient of each pixel of the original image P0) are the parameters of the original image mapping to the corrected image, the corrected image will have some pixels without corresponding original image mapping, which is manifested as the existence of background lines on the picture. Therefore, it is necessary to perform reverse mapping on the parameters calculated in the distortion iteration correction and the iteration splicing correction and use interpolation to eliminate the sawtooth. The original image P0 after the foregoing processing is projected on the screen, and multiple channels are projected simultaneously to measure the correction effect and the splicing effect.

[0066] The present method uses an iterative method to completely eliminate the position dependence between the projectors, the screen and the camera; uses a curve intersection correction algorithm to adapt to the projection imaging correction of a non-standard spherical screen; and uses an iteration correction method for splicing anchor points to realize full automation of splicing.

[0067] The test results on flight simulator show that the method has good correction ability, splicing ability and natural edge fusion effect. The correction can be completed within 2 hours and the theoretical splicing error is less than 1.4%. Using the traditional manual splicing method and skilled operation, it takes 4-5 hours to complete, and it cannot be guaranteed to succeed at one time. Compared with the traditional method, the automatic splicing correction method has the advantages of fast correction speed, low labor cost and high precision. In the future, we will strive to improve the projection resolution and correction accuracy.

Claims

1. An automatic curved surface projection correction and stitching method, characterized in that: Includes the following steps: Step 1: With all three projectors off and the UV lamps on, the three channels of the network... Each network camera takes a picture of the screen under its respective channel and saves an image containing the screen's marker points; Step 2: Process the screen calibration points to obtain clear screen calibration points; Step 3: Select the screen calibration points in an orderly manner; Step 4: Turn on the projector. The left and right channels will undergo iterative distortion correction first. The middle channel will undergo distortion correction after the other two channels have finished their corrections. Distortion correction uses an iterative correction method based on a curve intersection model. The method is as follows: For a region with M*N calibration points on the screen, fit M+N curves, with a total of (M+N)*4 parameters. Then, interpolate the parameters to infer multiple curves with the same number of pixels as the projected image. The transformed position of each pixel on the projected image P is then determined by the intersection of a horizontal and a vertical conic section. Step 5: After distortion correction is completed for all channels, each channel uses its own curve corner point xy coordinate data obtained in Step 4 to map the checkerboard pattern and project it onto the screen. A calibration map is generated using Apriltag for stitching the projected images of each channel. The middle channel projects the calibration map, while the left and right channels project the background color image. The left and right channels then identify the projection of the calibration map, and the results are combined to determine all the calibration anchor points of the stitching band. Then, a completely black image is projected from the middle channel, and the left and right channels are iteratively stitched and corrected. Step 6: Based on the trigonometric function-based color transition method, perform color transition processing on the splicing strip. The specific transition function used is: Among them, parameters A, B, and C are determined based on the continuity of the brightness of the transition zone after debugging; The coordinates of the splicing band are normalized; This indicates the brightness of the right channel splicing band during projection. This represents the theoretical sum of projected brightness; Step 7: Perform reverse mapping on the parameters calculated in the distortion iterative correction and iterative splicing correction and use interpolation to eliminate jagged edges.

2. The automatic curved surface projection correction and stitching method as described in claim 1, characterized in that: The manual calibration method is used to select the screen calibration points in an orderly manner. After the camera reads the image of the screen calibration points, the mouse is manually controlled to select a local image area containing a single calibration point. The position of the calibration point is automatically detected based on the selected local area and circled with a red circle. The selection order is to select row by row starting from the first available calibration point in the upper left corner. After all screen calibration points have been selected, the center coordinates of all calibrations are calculated, that is, the average of the x-coordinates and the average of the y-coordinates of all calibration points, and the data is saved in the required format.

3. The automatic curved surface projection correction and stitching method as described in claim 1, characterized in that: After turning on the projector in step 4, you need to synchronize all channels first.

4. The automatic curved surface projection correction and stitching method as described in claim 1, characterized in that: In step 5, AprilTag generates two calibration images, which are then projected onto the center channel.

5. The automatic curved surface projection correction and stitching method as described in claim 1, characterized in that: The iterative splicing correction method in step 5 is: the difference between the x-coordinate and y-coordinate of the splicing point and its corresponding anchor point. and Based on this distance difference, the position of the splicing point is iteratively modified. and Stop iterating when the time comes.

Images