A method, medium and system for projecting a special-shaped screen with multiple projectors and multiple control hosts

Abstract

The present invention provides a method, medium and system for projecting a special-shaped screen with multiple projectors of multiple control hosts, belonging to the technical field of special-shaped screen projection, and comprising: acquiring a monitoring image; establishing a spatial coordinate system and establishing a three-dimensional model of the special-shaped screen; positioning multiple projectors, and setting an initial projection area of ​​each projector on the three-dimensional model of the special-shaped screen; fitting the image to be projected on the projection surface of the three-dimensional model of the special-shaped screen; acquiring the overlapping part of the initial projection area of ​​each projector and the image to be projected, calculating the position adjustment of the projector, and marking the part of the overlapping projection overlapping area as the overlapping projection image range; controlling the connected projectors to project through the control host; calculating the brightness difference between the corresponding overlapping projection range and other areas on the basic image as the projection brightness difference; and performing a brightness reduction operation on the area of ​​the image to be projected corresponding to the overlapping projection image range until the brightness difference is eliminated.

Description

Technical Field

[0001] The present invention belongs to the technical field of special-shaped screen projection, and in particular, relates to a special-shaped screen projection method, medium and system for multi-control host and multi-projector. Background Art

[0002] With the continuous development of multimedia display technology, large-scale and large-size projection systems have been widely used. This system uses multiple projectors to work together to seamlessly splice multiple regional images together to form a large-area unified projection screen. It is widely used in public places such as conference rooms, exhibition halls, theaters, and amusement parks.

[0003] Different from the traditional single-plane projection screen, modern projection systems often need to project images onto non-planar shaped screens with complex geometric shapes to obtain more stunning visual effects. This shaped screen includes spherical, toroidal, concave and convex surfaces, etc., which can flexibly construct cylindrical projection, dome projection, panoramic projection and other projection methods to meet the needs of various application scenarios.

[0004] However, due to the complex and changeable geometric shapes of special-shaped screens, there are many technical challenges in achieving pixel-level seamless splicing. The existing common multi-projector splicing technologies can be roughly divided into the following methods:

[0005] (1) Multi-projection overlay technology based on edge blending: This technology adjusts the brightness of the overlapping projection areas to make multiple projection images transition smoothly in the edge areas, thereby achieving seamless splicing. However, since only the overlapping areas are processed, it is difficult to completely eliminate obvious splicing marks.

[0006] (2) Multi-projection stitching technology based on gamma correction: Using the gamma correction theory, the color gain, black level and other parameters of different projectors are adjusted so that the colors of multiple projected images can be more closely matched. However, it is still difficult to handle the brightness and darkness differences in local detail areas.

[0007] (3) Multi-projection stitching technology based on physical correction: By changing the position of the projector, its projection geometry is completely consistent with the projection screen, thus achieving seamless projection. However, the surface fitting accuracy of the special-shaped screen is low, and the projection distortion problem cannot be completely solved.

[0008] The above existing technologies all have the following common problems when applied to special-shaped screens:

[0009] (1) It is impossible to comprehensively consider the relative position relationship and projection area distribution of multiple projectors. Often, only a single projector is tuned in isolation, which makes it difficult to optimize the projection effect of the entire system.

[0010] (2) It is impossible to accurately analyze and correct the brightness differences in the overlapping projection areas, making it difficult to completely eliminate defects such as seams and light leakage, which affects the viewing quality. Summary of the invention

[0011] In view of this, the present invention provides a method, medium and system for projecting a multi-projector special-shaped screen with multiple control hosts, which can solve the technical problems in the prior art that, when projecting on a multi-projector special-shaped screen, the relative position relationship and projection area distribution of multiple projectors cannot be comprehensively considered and the brightness difference in the overlapping projection area cannot be accurately analyzed and corrected, resulting in poor projection effect.

[0012] The present invention is achieved in that:

[0013] A first aspect of the present invention provides a method for projecting a special-shaped screen with multiple projectors and multiple control hosts, which includes the following steps:

[0014] S01. Obtain monitoring images including all projectors and special-shaped screens through a binocular camera;

[0015] S02. Based on the monitoring image, a spatial coordinate system is established, and a three-dimensional model of the special-shaped curtain is established in the spatial coordinate system;

[0016] S03, positioning a plurality of projectors in the spatial coordinate system, and setting an initial projection area of ​​each projector on the three-dimensional model of the special-shaped screen according to projection parameters of the projectors;

[0017] S04, obtaining an image to be projected, and attaching the image to be projected to the projection surface of the three-dimensional model of the special-shaped curtain;

[0018] S05, obtaining the overlapped portion of the initial projection area of ​​each projector and the image to be projected as the initial projection image range of each projector;

[0019] S06, calculating the position adjustment of the projector to adjust the projection area of ​​each projector so as to eliminate the projection gap area and ensure that the projection overlap area is minimized;

[0020] S07, reacquiring the projection image range of each projector according to the adjusted projection area, and marking the portion of the overlapping projection overlap area as the overlapping projection image range;

[0021] S08, sending the image to be projected to the control host according to the projection image range of each projector, and controlling the connected projectors to project through the control host;

[0022] S09, obtaining the projection image on the special-shaped screen through the binocular camera as a basic image, and calculating the brightness difference between the corresponding overlapping projection range and other areas on the basic image as the projection brightness difference;

[0023] S10, based on the projection brightness difference, performing a brightness reduction operation on a region of the image to be projected corresponding to a range of the overlapping projection image;

[0024] S11, repeating steps S08 to S10 until the projection brightness difference decreases to a preset threshold.

[0025] On the basis of the above technical solution, the multi-control host multi-projector special-shaped screen projection method of the present invention can also be improved as follows:

[0026] The method for establishing the spatial coordinate system is as follows: detecting known geometric markers on the special-shaped curtain model, and using their positions and shapes to calculate the origin position and coordinate axis directions of the spatial coordinate system.

[0027] Among them, the step of fitting the image to be projected to the projection surface of the three-dimensional model of the special-shaped curtain is specifically: using a texture mapping algorithm to render and fit the content of the image to be projected to the surface of the three-dimensional model of the special-shaped curtain to generate a projection image model with a three-dimensional shape, or using a ray tracing algorithm to simulate the light propagation process and process shadows and reflection details; generating UV mapping coordinates of the special-shaped curtain model, and rendering the image to the model surface according to the UV coordinates.

[0028] Furthermore, when using the ray tracing algorithm, the light source parameters of the scene are set to at least include the light source type, position, intensity, and color; the material parameters are set to at least include reflectivity, refractive index, and roughness.

[0029] Among them, the binocular camera needs to be calibrated before collecting images, and the calibration process adopts a combination of Zhang Zhengyou calibration method and SPRT calibration method.

[0030] Furthermore, the step of calculating the posture adjustment of the projector is specifically: calculating the shortest distance between any two adjacent projection areas, if it is greater than a threshold, it is considered that there is a gap; calculating the overlapping area of ​​any two adjacent projection areas, if it is greater than a threshold, it is considered that there is too much overlap; constructing the minimum gap area and the minimum overlapping area as the objective function, taking the posture of each projector as the optimization variable, and using the random Monte Carlo method as the optimization algorithm to solve the projector posture combination that minimizes the objective function; recalculating the projection area of ​​each projector on the special-shaped screen according to the new posture, eliminating the gap area and reducing the overlapping area.

[0031] Furthermore, the step of reacquiring the projection image range of each projector according to the adjusted projection area and marking the part of the overlapping projection overlapping area as the overlapping projection image range is specifically: recalculating the projection area of ​​each projector on the special-shaped screen according to the adjusted new posture of the projector, and extracting the image of the corresponding area as the latest projection image range of each projector; traversing the set of triangular facets corresponding to each projection area, if the same facet is covered by multiple projection areas, the texture coordinate area corresponding to the facet is the overlapping projection image range; processing the projection area with occlusion relationship, and removing the completely occluded facet part from the occluded projection area image range.

[0032] Furthermore, the step of obtaining the projection image on the special-shaped screen as a basic image through the binocular camera and calculating the brightness difference between the corresponding overlapping projection range and other areas on the basic image as the projection brightness difference is specifically: collecting the three-dimensional image of the projection result as the basic image through the binocular camera; intercepting the overlapping projection image range part of the corresponding mark from the basic image as the overlapping area image, and intercepting the non-overlapping projection area part as the non-overlapping area image; calculating the difference between the average brightness value of the overlapping area image and the average brightness value of the non-overlapping area image as the projection brightness difference; setting multiple detection areas for the special-shaped screen, respectively calculating the brightness difference and taking the average value; considering the influence of ambient light, compensating the brightness difference calculation formula in real time.

[0033] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned multi-control host multi-projector special-shaped screen projection method.

[0034] A third aspect of the present invention provides a multi-control host multi-projector special-shaped screen projection system, which includes the above-mentioned computer-readable storage medium.

[0035] Compared with the prior art, the beneficial effects of the method, medium and system for projecting a special-shaped screen with multiple projectors and multiple control hosts provided by the present invention are as follows: the method for projecting a special-shaped screen with multiple projectors and multiple control hosts proposed by the present invention can effectively solve the technical defects of the prior art, such as the difficulty of seamless projection of the special-shaped screen, serious projection distortion and imperfect splicing, and comprehensively improve the quality and effect of the special-shaped screen projection, and have significant technical progress, which is mainly reflected in the following aspects:

[0036] (1) High-precision 3D modeling of special-shaped curtains of any geometric shape is achieved. By using 3D visual sensing technologies such as binocular cameras and structured light, the true 3D shape of the special-shaped curtain can be obtained quickly and efficiently.

[0037] (2) The overall optimization of the multi-projector layout is achieved. Unlike traditional technologies, the solution of the present invention does not perform local optimization on a single projector in isolation, but comprehensively considers the position relationship, projection parameters and other information of all projectors, and calculates the optimal mapping area of ​​each projector on the special-shaped screen under the premise of seamless coverage by solving the optimization objective function.

[0038] (3) Accurate detection of overlapping projection areas and quantification of brightness differences are achieved. The present invention collects projection result images in real time, compares the average brightness of overlapping areas and non-overlapping areas, and can accurately calculate the brightness difference between the two. Traditional technologies are often unable to accurately quantify this difference and can only be estimated through experience.

[0039] (4) A closed-loop automatic correction of brightness differences in overlapping areas is achieved. After obtaining the brightness difference, the present invention adjusts the brightness of the original image to be projected and repeats the closed-loop process of projection-detection-adjustment until the brightness difference is eliminated.

[0040] In summary, the technical solution of the present invention solves the technical problems in the prior art that, when projecting on a multi-projector special-shaped screen, the relative position relationship and projection area distribution of multiple projectors cannot be comprehensively considered and the brightness difference in the overlapping projection area cannot be accurately analyzed and corrected, resulting in poor projection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0042] Figure 1 A flow chart of the method provided by the present invention; DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0044] like Figure 1 As shown, it is a flow chart of a method for projection of a special-shaped screen with multiple projectors and multiple control hosts provided by the present invention. The method comprises the following steps:

[0045] S01. Obtain monitoring images including all projectors and special-shaped screens through a binocular camera;

[0046] S02. Based on the monitoring image, a spatial coordinate system is established, and a three-dimensional model of the special-shaped curtain is established in the spatial coordinate system;

[0047] S03, positioning multiple projectors in a spatial coordinate system, and setting an initial projection area of ​​each projector on the three-dimensional model of the special-shaped screen according to the projection parameters of the projectors;

[0048] S04, obtaining the image to be projected, and attaching the image to be projected to the projection surface of the three-dimensional model of the special-shaped screen;

[0049] S05, obtaining the overlapped portion of the initial projection area of ​​each projector and the image to be projected as the initial projection image range of each projector;

[0050] S06, calculating the position adjustment of the projector to adjust the projection area of ​​each projector so as to eliminate the projection gap area and ensure that the projection overlap area is minimized;

[0051] S07, reacquiring the projection image range of each projector according to the adjusted projection area, and marking the portion of the overlapping projection overlap area as the overlapping projection image range;

[0052] S08, sending the image to be projected to the control host according to the projection image range of each projector, and controlling the connected projectors to project through the control host;

[0053] S09, obtaining the projection image on the special-shaped screen through the binocular camera as the basic image, and calculating the brightness difference between the corresponding overlapping projection range and other areas on the basic image as the projection brightness difference;

[0054] S10, based on the projection brightness difference, performing a brightness reduction operation on the area of ​​the image to be projected corresponding to the overlapping projection image range;

[0055] S11, repeat steps S08 to S10 until the projection brightness difference decreases to a preset threshold.

[0056] The specific implementation methods of the above steps are described in detail below:

[0057] The specific implementation of step S01 is: obtain a monitoring image containing all projectors and special-shaped screens through a binocular camera. The purpose of this step is to obtain real-time visual information of the projection environment and provide basic data for subsequent steps. The implementation process is as follows:

[0058] First, you need to arrange the binocular camera so that it can capture all projectors and the entire special-shaped screen. The selection of the binocular camera needs to be determined according to the size of the actual scene to ensure sufficient resolution and field of view. The camera should be installed as high as possible and without obstruction to obtain a panoramic view.

[0059] Next, the binocular camera is calibrated using Zhang Zhengyou’s calibration method. Calibration images are taken at multiple viewing angles and positions using a calibration plate of known size, and features such as corner points are extracted to calculate the internal and external parameters of the binocular camera, including focal length, principal point, distortion coefficient, etc.

[0060] Then the binocular camera is started for real-time acquisition, the left and right camera images are corrected and dedistorted using calibration parameters, and the three-dimensional coordinates of each pixel are calculated based on the camera's parallax principle, ultimately generating a three-dimensional point cloud image containing all projectors and special-shaped screens.

[0061] When acquiring the 3D point cloud image, it is necessary to filter and segment the point cloud image, remove irrelevant background points, and only retain the point cloud data of the projector and the special-shaped screen. The most commonly used filtering algorithms are statistical filtering and radius filtering, which extract the special-shaped screen plane based on the curvature and other features of the points. The segmentation algorithm can use methods such as clustering based on region growing and RANSAC segmentation based on model fitting. The threshold setting needs to be adjusted according to the specific scene.

[0062] Through the above steps, you can obtain monitoring images including all projectors and special-shaped screens, laying the foundation for subsequent steps.

[0063] The specific implementation of step S02 is: based on the monitoring image, a spatial coordinate system is established, and a three-dimensional model of the special-shaped curtain is established in the spatial coordinate system. The function of this step is to construct a three-dimensional geometric model of the special-shaped curtain, providing a basis for determining and adjusting the projection area. The implementation process is as follows:

[0064] First, the area of ​​the special-shaped curtain needs to be determined in the 3D point cloud monitoring image. The special-shaped curtain point cloud can be automatically extracted by manual marking or using the point cloud segmentation algorithm. Next, the special-shaped curtain point cloud is meshed to generate a mesh model composed of triangular facets.

[0065] Since the original point cloud has noise and missing, the point cloud needs to be resampled and smoothed when meshing. Resampling algorithms include moving cubes, moving spheres, etc., which can obtain evenly distributed point clouds. Smoothing algorithms can use moving least squares surfaces, bicubic B-splines, etc. to eliminate sharp protrusions and depressions. Appropriate parameters need to be set according to the complexity of the special-shaped curtain.

[0066] Next, we need to establish a spatial coordinate system, and the special-shaped curtain model will be positioned with reference to this coordinate system. Usually, the spatial coordinate system is determined with a corner point of the special-shaped curtain as the origin, a certain edge of the special-shaped curtain as the X-axis, and the surface normal vector as the Z-axis. The coordinate system can also be automatically calculated using known geometric markers on the special-shaped curtain.

[0067] Then the gridded special-shaped curtain point cloud is converted into a spatial coordinate system, and a closed triangular mesh model is generated using methods such as Poisson surface reconstruction and marching cube algorithm, which is the final three-dimensional model of the special-shaped curtain. This model can accurately reflect the actual shape and size of the special-shaped curtain.

[0068] After completing the above steps, a three-dimensional geometric model of the special-shaped screen based on the spatial coordinate system is obtained, laying the foundation for the subsequent arrangement and adjustment of the projection area.

[0069] The specific implementation of step S03 is: positioning multiple projectors in the spatial coordinate system, and setting the initial projection area of ​​each projector on the three-dimensional model of the special-shaped screen according to the projection parameters of the projector. The function of this step is to determine the projection position of each projector on the special-shaped screen, in preparation for combining and splicing into a complete image. The implementation process is as follows:

[0070] According to the installation position of the projector, the point cloud cluster corresponding to each projector is extracted in the 3D monitoring image, and the position and orientation of each projector in the spatial coordinate system are estimated. Here, the cylindrical shape of the point cloud cluster can be fitted using algorithms such as spherical Hough transform and RANSAC to obtain its position and orientation vector.

[0071] After determining the position and orientation of each projector, the next step is to obtain the projection parameters of each projector, including field of view, resolution, projection distance, etc. These parameters can be found in the projector manual or obtained through calibration.

[0072] Then, according to the position, orientation and projection parameters of the projector, the viewing cone of each projector is calculated, and the viewing cone is intersected with the three-dimensional model of the special-shaped screen to determine the projectable area on the special-shaped screen, which is the initial projection area of ​​each projector.

[0073] It should be noted that due to the irregular shape of the special-shaped screen, the initial projection areas of multiple projectors are likely to overlap and have gaps, so adjustments need to be made later. In addition, the projection parameters of the projector will also affect the size and shape of the projection area.

[0074] When setting the initial projection area, you also need to consider the occlusion problem between projectors. Based on the spatial geometric position relationship, you can determine whether a projector will be blocked by other projectors or scene objects. If there is occlusion, you need to adjust the projection area.

[0075] After completing the above steps, the position and orientation of each projector in space are obtained, and their initial projection areas on the special-shaped screen are determined, which prepares for the subsequent completion of the projection mapping of the entire special-shaped screen.

[0076] The specific implementation of step S04 is: obtaining the image to be projected, and fitting the image to be projected on the projection surface of the three-dimensional model of the special-shaped curtain. The function of this step is to map the image content to be projected onto the three-dimensional model of the special-shaped curtain, so as to prepare for subsequent segmentation and processing. The implementation process is as follows:

[0077] First, you need to obtain the image to be projected, which can be a preset image selected from the image library or an image frame captured by a camera in real time. For the image frame captured in real time, you need to remove the background and only keep the target content.

[0078] Next, the content of the image to be projected is rendered onto the surface of the 3D model of the special-shaped screen to form a projection image model with a 3D shape. Rendering algorithms include texture mapping and ray tracing. Texture mapping is to fit the entire 2D image onto the surface of the 3D model. It is fast and can maintain the original resolution. Ray tracing simulates the process of light propagation in a 3D scene. It can handle details such as shadows and reflections well, but the amount of calculation is large.

[0079] When applying texture mapping, you need to first unfold the 3D model of the special-shaped curtain to a 2D plane to generate UV mapping coordinates. Then render the image onto the model surface according to the UV coordinates. There are many options for unfolding algorithms, such as equidistant array projection, minimum stretching and other deformation algorithms, and you need to balance projection distortion and texture resolution.

[0080] When applying ray tracing, you need to set parameters such as scene light source and material properties to simulate the real lighting effect of the 3D scene. The large-scale parallel computing capability of GPU can be used to accelerate the calculation.

[0081] After the mapping is completed, the surface of the 3D model of the special-shaped screen contains the image content to be projected, which is ready for the next step of calculating the image area corresponding to each projector.

[0082] The specific implementation of step S05 is: obtaining the overlapped part of the initial projection area of ​​each projector and the image to be projected as the initial projection image range of each projector. The function of this step is to divide the overall projection content into multiple parts and assign them to different projectors, laying the foundation for collaborative projection. The implementation process is as follows:

[0083] Based on the initial projection area of ​​each projector determined in the previous step, it is intersected with the three-dimensional model of the special-shaped screen with the projection image attached to extract the projection image area corresponding to each projector. The specific method is to correspond the initial projection area to the triangular mesh of the special-shaped screen model and extract all the triangular facets covered by the area.

[0084] Then, we traverse these triangular patches, read the corresponding texture coordinates, and extract the corresponding image area from the original image to be projected, which is the initial projection image range of the projector. Here, we need to pay attention to the interpolation problem of texture coordinates at the image boundary to avoid gaps.

[0085] Due to the overlap of projection areas, the projection image ranges of different projectors may overlap. Such overlapping areas will be marked as overlapping projection image ranges that require brightness adjustment. To determine whether there is overlap, it is possible to detect whether multiple triangles use the same texture coordinates.

[0086] For areas on the 3D model of the special-shaped curtain that are not covered by any projection area, the corresponding image part will be clipped and will not participate in the final projection. This may happen at the edge of the special-shaped curtain.

[0087] Finally, the initial projection image range corresponding to each projector was obtained, and the overlapping projection image range that required brightness adjustment was marked, preparing for subsequent collaborative projection adjustment and final projection.

[0088] Specifically, let the image to be projected be I, the three-dimensional model of the special-shaped screen be M, and the initial projection area of ​​the i-th projector be R i .

[0089] R i Intersect with M with projection image I, extract all R i The set of covered triangle patches T i :

[0090] T i ={t|t∈M∩R i};

[0091] Among them, t is a triangular patch.

[0092] Traverse T i For each patch t in I, read its corresponding texture coordinates (u, v), and cut the corresponding area from I as the initial projection image range P of the i-th projector i :

[0093] P i ={p|(u, v)∈t, p=I(u, v)};

[0094] Check if the same patch t is represented by multiple R i Coverage, if so, the corresponding (u, v) area is marked as the overlapping projection image range O:

[0095]

[0096] For M not covered by any Ri The covered area is cut off from the part corresponding to I.

[0097] The specific implementation of step S06 is: calculate the position adjustment of the projector to adjust the projection area of ​​each projector so as to eliminate the projection gap area and ensure that the projection overlap area is minimized. The function of this step is to optimize the projection layout of multiple projectors, remove the gaps and redundant overlaps between projections, and prepare for obtaining a complete and seamlessly spliced ​​projection image. The implementation process is as follows:

[0098] Based on the initial position of each projector determined in step S03 and the initial projection area obtained in step S05, the projection area is fine-tuned and optimized. First, determine whether there is a projection gap, calculate the shortest distance between any two adjacent projection areas, and if the distance is greater than a certain threshold, it is considered that there is a gap. At the same time, determine whether there are too many overlapping areas, calculate the overlapping area of ​​any two adjacent projection areas, and if the area is greater than the set threshold, it is considered that there is too much overlap. These two thresholds can be set empirically based on factors such as the size of the special-shaped screen and the number of projectors.

[0099] Next, we construct the optimization objective function, one of which is to minimize the gap area and the other is to minimize the overlap area. We use the position and orientation of each projector (including position and orientation) as the optimization variable, and use combinatorial optimization algorithms such as random Monte Carlo method and simulated annealing algorithm to solve the projector position combination that minimizes the objective function without changing the number of projectors.

[0100] During the optimization process, it is necessary to ensure that the adjusted projector position and orientation are within the feasible range, that is, there should be no occlusion with the special-shaped screen, and it should not deviate from the site boundary, etc. At the same time, the shortest projection distance of the projector to the special-shaped screen should be kept within a reasonable range to avoid imaging distortion caused by being too close or too far. The corresponding penalty term can be added to the objective function to constrain the optimization.

[0101] When the optimization algorithm converges, the new position of each projector can be obtained. At this time, the projection area of ​​each projector on the special-shaped screen is recalculated based on the new position. Ideally, each projection area will be seamlessly spliced ​​with minimal overlap.

[0102] Specifically, the projected gap area is defined as:

[0103]

[0104] Among them, d(p i , p j ) is p i and p j The Euclidean distance of s is the gap threshold, is an indicative function.

[0105] The projection overlap area is defined as:

[0106]

[0107] Among them, ∈ o is the overlap threshold.

[0108] Construct the objective function:

[0109] f(X)=w s S+w o O;

[0110] Among them, X is the set of all projector poses, w s and w o is the weight coefficient.

[0111] The simulated annealing algorithm is used to solve Get the adjusted projector pose set X * , recalculate each projection area

[0112] After completing the above steps, the adjusted projection layout plan was obtained, ready for high-quality seamless projection mapping on the special-shaped screen.

[0113] The specific implementation of step S07 is: according to the adjusted projection area, re-acquire the projection image range of each projector, and mark the part that overlaps the projection overlap area as the overlapping projection image range. The function of this step is to update the projection image range corresponding to each projector to prepare for processing the projection overlap area. The implementation process is as follows:

[0114] Based on the new position of the projector optimized in the previous step, the projection area corresponding to each projector on the special-shaped screen is recalculated according to the method of step S05. Then, the images corresponding to these areas are extracted as the latest projection image range of each projector.

[0115] Next, we determine and mark the overlapping projection areas between projectors by traversing the triangle patch set corresponding to each projection area and finding whether the same patch is covered by multiple projection areas. If so, the texture coordinate area corresponding to the patch is the overlapping projection image range.

[0116] For the projection areas with occlusion, further processing is required. Check whether there is a face on the 3D model of the special-shaped curtain that is completely occluded, that is, the face is only covered by one projection area. If so, remove it from the image range of the occluded projection area to ensure that there is no occlusion area during the final splicing.

[0117] Specifically, based on X * and Repeat step S05 to obtain the latest projection image range of each projector and overlap projection image range O * .

[0118] For the projection area with occlusion Check whether there is a patch t that is only covered by a single Cover, if so, from Remove the corresponding part:

[0119]

[0120] After completing the above steps, not only the final projection image range of each projector is obtained, but also the overlapping projection image range that needs brightness adjustment is marked, preparing for the next step of distributing the image through the control host.

[0121] The specific implementation of step S08 is: according to the projection image range of each projector, the image to be projected is sent to the control host, and the control host controls the connected projectors to project. The function of this step is to distribute the segmented image to each projector to realize the overall projection of the special-shaped screen. The implementation process is as follows:

[0122] First, you need to build a network connection between the control host of the projection system and multiple projection slaves. You can use a wired network such as Ethernet or a wireless network. The control host needs to be installed with image processing and communication software, and the projection slaves need to be able to receive instructions from the control host.

[0123] Then, the control host cuts the original whole image to be projected according to the final projection image range of each projector obtained in step S07 to generate corresponding sub-images, and sends the sub-images to the corresponding projection slaves through the network.

[0124] After receiving the sub-image, the projection slave decodes and processes the image, and inputs the image to the projector for projection display. It should be noted that each projection slave must be able to accurately control the position and projection transformation parameters of the received sub-image to ensure that they can be correctly and seamlessly spliced ​​on the special-shaped screen. The calibration of the projection slave and the projection area can be completed in advance.

[0125] Through the real-time control of the host computer, the synchronization of the entire projection process can be ensured, so that multiple projectors can work together to complete the overall projection mapping of the special-shaped screen. In real-time control, it is necessary to deal with problems such as network packet loss and image transmission delay to ensure smoothness.

[0126] Specifically, the control host cuts I into sub-images I i :

[0127]

[0128] The data are sent to the corresponding projection slaves through the network, and the slaves decode and control the projector to project and display.

[0129] After completing this step, you can see that multiple projectors work together to seamlessly project the overall projection content onto the special-shaped screen with high quality, preparing for the subsequent correction of brightness differences.

[0130] The specific implementation of step S09 is: the binocular camera obtains the projection image on the special-shaped screen as the basic image, and calculates the brightness difference between the corresponding overlapping projection range and other areas on the basic image as the projection brightness difference. The purpose of this step is to detect whether there is brightness inconsistency in the overlapping area of ​​the projection image, and prepare for brightness correction. The implementation process is as follows:

[0131] After the overall projection is achieved in the previous step, the binocular camera needs to be used to collect the projection result image in real time as the basic image for subsequent brightness difference detection and correction. Since the binocular camera can obtain three-dimensional information, the surface shape and texture details of the special-shaped curtain can be well reconstructed.

[0132] The portion corresponding to the overlapping projection image range marked in step S07 is cut out from the basic image as the overlapping projection area image, and the portion of the non-overlapping projection area is cut out as the non-overlapping area image.

[0133] Compare the average brightness value of the overlapping area image with the average brightness value of the non-overlapping area image to obtain the brightness difference value, which is the current projection brightness difference. You can set multiple detection areas for the special-shaped screen, calculate the brightness difference respectively, and take the average value as the final value of the projection brightness difference.

[0134] In addition, the influence of ambient light needs to be considered. Light sensors can be set around the special-shaped screen to detect changes in ambient light intensity in real time. When a large change in ambient light is detected, the calculation formula for the brightness difference should be compensated in real time to eliminate the environmental impact.

[0135] The purpose of calculating the brightness difference is to detect whether there is a significant brightness inconsistency area. When the projection brightness difference is greater than a certain threshold, it is considered that brightness correction is required. The threshold can be pre-set according to the actual scene and requirements, or it can be adaptively adjusted through machine learning methods during system operation.

[0136] Specifically, a binocular camera is used to collect the projection result 3D image I′, from which the overlapping area image I′ is intercepted. O and non-overlapping region image I′ N :

[0137]

[0138]

[0139] Calculate the average brightness of the two and The projection brightness difference is:

[0140]

[0141] Set K detection areas for the special-shaped curtain, and the brightness difference of the kth area is ΔI k , then the final projection brightness difference is:

[0142]

[0143] Considering the influence of ambient light on L, make real-time corrections:

[0144] ΔI ′ =ΔI+f(L);

[0145] Among them, f is the correction function, which needs to be obtained by pre-fitting.

[0146] After completing this step, the brightness difference between the overlapping area and the non-overlapping area during the current multi-projector collaborative projection can be obtained, laying the foundation for subsequent brightness correction.

[0147] The specific implementation of step S10 is: based on the projection brightness difference, the brightness of the area corresponding to the overlapping projection image range of the image to be projected is reduced. The function of this step is to eliminate the brightness inconsistency of the overlapping area by adjusting the image brightness and improve the overall projection effect. The implementation process is as follows:

[0148] Based on the current projection brightness difference calculated in step S09, it is determined whether the brightness difference exceeds an acceptable threshold range. If it is within the threshold range, no brightness adjustment is required; if it exceeds the threshold range, brightness correction is required.

[0149] The basic idea of ​​brightness correction is to appropriately reduce the brightness value of the overlapping projection area so that its brightness value gradually approaches the brightness value of the non-overlapping area, and finally make the average brightness values ​​of the overlapping area and the non-overlapping area consistent.

[0150] The specific method is as follows: first, return to the original whole image to be projected, and use the marking information of the overlapping projection image range obtained in step S07 to extract the corresponding overlapping area image. Then, according to the size of the current projection brightness difference, set an attenuation coefficient to attenuate the pixel value of the overlapping area image so that its average brightness value decreases.

[0151] The attenuation algorithm can adopt simple linear attenuation, or nonlinear Gaussian attenuation, logarithmic attenuation, etc. Select a suitable algorithm and adjust the parameters to make the change of brightness value relatively smooth and avoid the loss of color level.

[0152] At the same time, for the junction of overlapping areas and non-overlapping areas, interpolation and smoothing processing is required to make the transition of brightness values ​​more natural and avoid obvious brightness splicing traces. Algorithms such as bicubic convolution kernels can be used for edge smoothing.

[0153] After adjusting the brightness, the processed image is synthesized into a new image to be projected, replacing the original image to be projected, and entering the next round of projection cycle.

[0154] During the brightness correction process, a certain convergence margin needs to be reserved, that is, the projection brightness difference will not be 0 immediately after each adjustment, but will gradually converge to avoid repeated jitters caused by over-correction. The size of the convergence margin can be set to a few percentage points of the brightness difference in the previous stage.

[0155] Specifically, let the attenuation coefficient be α, and for I O The pixel value p is attenuated:

[0156]

[0157] Use bicubic convolution kernel K for edge smoothing:

[0158] p s =p*K;

[0159] Order I new =I′ N ∪{p′|p′∈I′ O}, the processed I new Replace I and enter the next projection cycle.

[0160] The attenuation coefficient α and the smoothing kernel K need to be optimized according to the actual situation. A dynamic step size η can be introduced and the maximum and minimum values ​​can be set:

[0161]

[0162] Among them, γ and β are preset parameters used to control the step update speed and size.

[0163] After completing this step, the brightness value of the overlapping area of ​​the image to be projected will gradually decrease until it is basically consistent with the brightness value of the non-overlapping area, thereby effectively eliminating the brightness splicing problem that affects the overall projection effect.

[0164] The specific implementation of step S11 is: repeat steps S08 to S10 until the projection brightness difference reaches a preset threshold. The function of this step is to make the brightness value of the overlapping area consistent with the brightness value of the non-overlapping area through cyclic iteration, so as to obtain a high-quality seamless overall projection effect. Among them, the preset threshold can be set according to the operator's senses, or it can be set to 10% of the maximum brightness; the specific implementation process is as follows:

[0165] First, the new image to be projected obtained in step S10 is sent to the control host, and the control host controls each projector to perform projection display in the manner of step S08, that is, the projection step of S08 is completed again.

[0166] Then, step S09 is entered to collect a new projection result image through the binocular camera, and a new projection brightness difference is calculated based on the image. It is determined whether the new projection brightness difference is 0 or within an acceptable small range. If so, the brightness correction is terminated to complete the entire multi-projector special-shaped screen projection process; if not, step S10 is entered to perform a new round of brightness adjustment on the image.

[0167] In the new round of brightness adjustment, the brightness attenuation coefficient and the parameters of the attenuation algorithm should be dynamically adjusted in combination with the correction effect of the previous round, in order to obtain a faster convergence speed. For example, a dynamic step size can be introduced, and the step size can be appropriately enlarged when the correction effect of the previous round is good, and vice versa, the step size can be reduced to balance the convergence speed and brightness smoothness.

[0168] During the entire iterative cycle of brightness correction, the system can selectively perform other image enhancement processing, such as denoising, sharpening, etc., to further improve the projection effect.

[0169] This iterative cycle will continue until the projection brightness difference drops to a smaller acceptable range, thus achieving seamless splicing projection with completely consistent overlapping and non-overlapping areas. At this point, the goal of the entire multi-projector special-shaped screen projection system is achieved.

[0170] It should be noted that the variables involved in the calculation process in the specific implementation of the present invention are explained as shown in the following table:

[0171] Table 1 Variable explanation table

[0172]

[0173]

[0174]

[0175] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned multi-control host multi-projector special-shaped screen projection method.

[0176] A third aspect of the present invention provides a multi-control host multi-projector special-shaped screen projection system, which includes the above-mentioned computer-readable storage medium.

[0177] Specifically, the present invention firstly uses a binocular camera or other three-dimensional visual sensor to capture a three-dimensional monitoring image containing all projectors and the special-shaped screen, and uses computer vision technologies such as point cloud processing and surface reconstruction to perform three-dimensional modeling of the special-shaped screen to accurately obtain its true geometric shape. This step solves the problem of being unable to accurately describe the shape of the special-shaped screen.

[0178] Next, the present invention combines the obtained 3D model of the special-shaped screen with the position, orientation and projection parameters of each projector, and calculates the optimal mapping area of ​​multiple projectors on the special-shaped screen through spatial geometric deduction and optimization algorithm adjustment, so that the entire special-shaped screen surface can be completely and seamlessly covered by the projectors, and the overlapping projection area is minimized. This step solves the problem of not being able to fully consider the position relationship of each projector.

[0179] After obtaining the mapping areas of each projector on the special-shaped screen, the present invention divides the original image to be projected according to these areas, and distributes the corresponding image to each projector through the network to complete the preliminary projection mapping of the entire special-shaped screen. At this time, there may be a problem of inconsistent brightness in the overlapping areas.

[0180] The scheme and principle of the present invention can be summarized into the following two key points:

[0181] (1) 3D modeling and geometric fitting: 3D modeling can accurately obtain the actual shape of the special-shaped screen, while geometric fitting can optimize the projector layout, fundamentally solving the various projection distortions and defects caused by inaccurate shape description or local optimization in traditional technologies.

[0182] (2) Brightness difference correction feedback control: By acquiring the projection results in real time, analyzing the brightness difference in the overlapping area, and adjusting the brightness of the original image, a closed-loop control is formed. In theory, the stitching defects in the overlapping area can be completely eliminated, achieving true pixel-level seamless stitching.

[0183] The following is a specific embodiment of the present invention:

[0184] This embodiment is a spherical special-shaped screen projection system established in a certain science and technology exhibition hall based on the solution of the present invention. The system is used to project ultra-high-definition panoramic spherical screen images in the zenith area of ​​the exhibition hall to provide an immersive visual experience for the audience.

[0185] The main body of the spherical special-shaped screen is a large spherical space with a radius of 20 meters, and the inner wall is covered with high-reflectivity projection material. The special-shaped screen is composed of 20 fan-shaped curved surface pieces in the shape of an icosahedron, and the size of each curved surface piece is about 10 meters × 15 meters. There is also a circular area with a radius of 5 meters at the bottom as the audience activity area.

[0186] The projection system is composed of 17 4K high-definition projectors, 16 of which are installed around the top of the sphere to cover the main body of the sphere, and one projector is installed at the center of the sphere facing upwards to cover the bottom annular area.

[0187] In order to obtain an accurate 3D model of the special-shaped curtain, the system uses a ZED binocular depth camera installed in the center of the sphere. The camera has a horizontal field of view of 90 degrees, a resolution of 3840×1080, and a maximum range of 20 meters. The mobile calibration method is used to calibrate the camera to obtain its internal and external parameters.

[0188] Internal parameters:

[0189] Camera matrix:

[0190] Distortion coefficient:

[0191] k1=-0.125, k2=0.064, p1=0.0014, p2=-0.00067

[0192] External parameters:

[0193] Translation vector: T = [0 0 0]

[0194] Rotation matrix: After calibration, the calibration accuracy can reach 0.05 pixels by using the optimized combination of Zhang Zhengyou calibration method and SPRT algorithm.

[0195] Next, start the binocular camera to scan the environment. Set the point cloud resolution to 2cm, the statistical filter radius to 3cm, and the RANSAC plane segmentation threshold to 10cm. After a series of processing, the point cloud data of 20 curved surface patches of the special-shaped curtain are successfully obtained, totaling 2253451 points.

[0196] After filtering and smoothing the patch point cloud, the moving cube algorithm is used for resampling, and the resampling interval is 2 cm. The smoothing algorithm uses the moving least squares surface method, with a window radius of 5 cm and 15 nearest neighbor samples.

[0197] When constructing the spatial coordinate system, the center of the sphere is set to the origin (0, 0, 0), and a known circular marker is detected at (0, 18.7, 3.5) with a radius of 2 meters, thereby determining that the Z axis is vertically upward and the X axis is perpendicular to the vector from the origin to the circular marker.

[0198] Finally, a Poisson surface reconstruction model of 20 curved facets of the special-shaped curtain is generated, and the number of triangular facets is 5652834. The relative error between the overall model of the special-shaped curtain and the actual measured size is only 0.06%.

[0199] Based on the three-dimensional model of the special-shaped screen and combined with the actual positions and parameters of 17 projectors, their optimal projection areas on the special-shaped screen were calculated.

[0200] The overlap threshold is set to: ∈ o =1m 2

[0201] The gap threshold is set to: ∈ s =0.5m

[0202] Optimize the objective function weight coefficient:

[0203] w s =10 4 , w o =10 3

[0204] The simulated annealing method was used to solve the problem, with an initial temperature of 1000, an end temperature of 0.01, a drop rate of 0.95, and a maximum number of iterations of 10,000. The optimization results show that each projection area completely covers the special-shaped screen surface, and the maximum overlapping area is 0.63m 2 There are 14 gaps, and the maximum gap distance is 0.35 meters.

[0205] Afterwards, the mapping areas of the 17 projectors on the special-shaped screen were allocated according to the optimization results. The number of triangle facets corresponding to the mapping area of ​​a certain projector is 172,546 in total, and the projected image size is 3080×1744 pixels.

[0206] The control host distributes the original 8K resolution (7680×4320) panoramic images to 17 projection slaves according to these mapping areas, realizing the preliminary projection mapping of the entire special-shaped screen. At this time, the binocular camera is used to obtain the basic image of the projection result, and it is found that there is a certain brightness difference in the overlapping area.

[0207] Four detection areas are set, the average brightness of the overlapping area in area 1 is 137.2, the average brightness of the non-overlapping area is 118.3, and the brightness difference ΔI1=18.9;

[0208] The average brightness of the overlapping area in zone 2 is 125.6, and that of the non-overlapping area is 116.8, with a brightness difference of ΔI2 = 8.8;

[0209] The average brightness of the overlapping area in zone 3 is 141.5, and that of the non-overlapping area is 120.1, with a brightness difference of ΔI3 = 21.4;

[0210] The average brightness of the overlapping area in zone 4 is 126.7, and that of the non-overlapping area is 113.6, with a brightness difference of ΔI4 = 13.1;

[0211] The average brightness difference ΔI=15.55.

[0212] Assume that the ambient light effect term f(L) = 0.005L 2 +0.3L, the current ambient brightness L=520lux, then the corrected brightness difference is:

[0213] ΔI′=ΔI+f(L)=15.55+0.005×520 2 +0.3×520=27.75

[0214] Set the threshold ΔI th =5, the current brightness difference has exceeded the threshold and brightness adjustment is required. The brightness attenuation coefficient α is taken as 0.15, and the corresponding brightness overlapping area is adjusted. The edge of the overlapping area is smoothed using a bicubic convolution kernel of size 7×7.

[0215] The adjusted panoramic image is re-input and projected again. A new projection result image is obtained, and the average brightness difference is reduced to ΔI = 10.32, which still exceeds the threshold, and a new round of brightness adjustment is entered.

[0216] The dynamic step size update strategy is applied, with a priori γ = 0.1, initial step size η0 = 0.2, and upper and lower bounds η min =0.05,η max =0.5.

[0217] According to the last two adjustment results: 15.55→10.32, the step size adjustment is:

[0218] Δη=0.1×(10.32-15.55)=-0.523

[0219] So the new step size η new =0.2-0.523=-0.323, take the upper limit 0.5.

[0220] New attenuation coefficient α new =0.5×0.5=0.25

[0221] Similarly, after 12 rounds of adjustment, the final brightness difference dropped to 2.37, which is below the threshold range, thus achieving an ideal seamless splicing projection effect. The data statistics of the entire adjustment process are as follows:

[0222]

[0223]

[0224] Through the above 12 rounds of closed-loop control brightness adjustment, the brightness difference in the overlapping area was successfully eliminated, and high-quality seamless splicing of special-shaped screen panoramic projection was achieved.

[0225] This embodiment fully verifies the feasibility and superiority of the solution of the present invention, and the specific data also reflects the high precision and stability of the inventive method.

[0226] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for projecting a special-shaped screen with multiple projectors and multiple control hosts, characterized in that: The following steps are involved: S01. Obtain monitoring images including all projectors and special-shaped screens through a binocular camera; S02. Based on the monitoring image, a spatial coordinate system is established, and a three-dimensional model of the special-shaped curtain is established in the spatial coordinate system; S03, positioning a plurality of projectors in the spatial coordinate system, and setting an initial projection area of ​​each projector on the three-dimensional model of the special-shaped screen according to projection parameters of the projectors; S04, obtaining an image to be projected, and attaching the image to be projected to the projection surface of the three-dimensional model of the special-shaped curtain; S05, obtaining the overlapped portion of the initial projection area of ​​each projector and the image to be projected as the initial projection image range of each projector; S06, calculating the position adjustment of the projector to adjust the projection area of ​​each projector so as to eliminate the projection gap area and ensure that the projection overlap area is minimized; S07, reacquiring the projection image range of each projector according to the adjusted projection area, and marking the portion that overlaps the projection overlap area as the overlap projection image range; S08, sending the image to be projected to the control host according to the projection image range of each projector, and controlling the connected projectors to project through the control host; S09, obtaining the projection image on the special-shaped screen through the binocular camera as a basic image, and calculating the brightness difference between the corresponding overlapping projection range and other areas on the basic image as the projection brightness difference; S10, based on the projection brightness difference, performing a brightness reduction operation on a region of the image to be projected corresponding to a range of the overlapping projection image; S11, repeating steps S08 to S10 until the projection brightness difference decreases to a preset threshold; Among them, the step of calculating the posture adjustment of the projector is specifically: calculating the shortest distance between any two adjacent projection areas, if it is greater than a threshold, it is considered that there is a gap; calculating the overlapping area of ​​any two adjacent projection areas, if it is greater than a threshold, it is considered that there is too much overlap; constructing the minimum gap area and the minimum overlapping area as the objective function, taking the posture of each projector as the optimization variable, and using the random Monte Carlo method as the optimization algorithm to solve the projector posture combination that minimizes the objective function; recalculating the projection area of ​​each projector on the special-shaped screen according to the new posture, eliminating the gap area and reducing the overlapping area.

2. The method for projecting a special-shaped screen with multiple projectors and multiple control hosts according to claim 1, characterized in that: The method of establishing the spatial coordinate system is: detecting the known geometric markers on the special-shaped curtain model, and using their positions and shapes to calculate the origin position and coordinate axis directions of the spatial coordinate system.

3. The method for projecting a special-shaped screen with multiple projectors and multiple control hosts according to claim 1, characterized in that: The step of attaching the image to be projected to the projection surface of the three-dimensional model of the special-shaped curtain specifically comprises: using a texture mapping algorithm to render and attach the image content to the surface of the three-dimensional model of the special-shaped curtain to generate a projection image model with a three-dimensional shape, or using a ray tracing algorithm to simulate the light propagation process and process shadow and reflection details; Generate UV mapping coordinates of the special-shaped curtain model and render the image to the model surface according to the UV coordinates.

4. The method for projecting a special-shaped screen with multiple projectors and multiple control hosts according to claim 3, characterized in that: When using the ray tracing algorithm, the light source parameters of the scene are set to at least include the light source type, position, intensity, and color; the material parameters are set to at least include reflectivity, refractive index, and roughness.

5. The method for projection of a special-shaped screen with multiple projectors and multiple control hosts according to claim 1, characterized in that: The binocular camera needs to be calibrated before collecting images, and the calibration process adopts a combination of Zhang Zhengyou calibration method and SPRT calibration method.

6. A method for projecting a special-shaped screen with multiple projectors using multiple control hosts according to claim 5, characterized in that: The steps of reacquiring the projection image range of each projector according to the adjusted projection area and marking the part of the overlapping projection overlapping area as the overlapping projection image range are specifically: recalculating the projection area of ​​each projector on the special-shaped screen according to the adjusted new posture of the projector, and extracting the image of the corresponding area as the latest projection image range of each projector; traversing the set of triangular facets corresponding to each projection area, if there is a same facet covered by multiple projection areas, then the texture coordinate area corresponding to the facet is the overlapping projection image range; processing the projection area with an occlusion relationship, and removing the completely occluded facet part from the occluded projection area image range.

7. The method for projecting a special-shaped screen with multiple projectors and multiple control hosts according to claim 6, characterized in that: The step of obtaining the projection image on the special-shaped screen as the basic image through the binocular camera and calculating the brightness difference between the corresponding overlapping projection range and other areas on the basic image as the projection brightness difference is specifically: collecting the three-dimensional image of the projection result as the basic image through the binocular camera; intercepting the overlapping projection image range part corresponding to the mark from the basic image as the overlapping area image, and intercepting the non-overlapping projection area part as the non-overlapping area image; The difference between the average brightness value of the image in the overlapping area and the average brightness value of the image in the non-overlapping area is calculated as the projection brightness difference; multiple detection areas are set for the special-shaped screen to calculate the brightness difference and take the average value; considering the influence of ambient light, the brightness difference calculation formula is compensated in real time.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the multi-control host multi-projector special-shaped screen projection method according to any one of claims 1 to 7.

9. A multi-control host multi-projector special-shaped screen projection system, characterized in that: The computer-readable storage medium of claim 8 is included, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, the method for projecting a multi-projector special-shaped screen with multiple control hosts is used to execute the method according to any one of claims 1 to 7; the system also includes a control host, 17 projectors and a binocular depth camera.

Images