An automatic correction method and system for fused projection

Abstract

This invention discloses an automatic correction method and system for fused projection, comprising: modeling the receiving surface of the projection to obtain a receiving surface model; constructing a pixel mapping between camera space and projector space, and a pixel mapping between camera space and the receiving surface model space; obtaining a pixel correspondence between the receiving surface model and the projector space based on the pixel mappings of the camera space, projector space, and receiving surface model space; and mapping the content to be projected onto the projector space according to the pixel correspondence. This invention, based on the pixel mappings of camera space, projector space, and receiving surface model space, can obtain the pixel correspondence between the receiving surface model and the projector space; therefore, the content to be projected can be mapped onto the projector space according to the pixel correspondence, realizing automatic correction during multi-camera fused projection, greatly improving the efficiency and accuracy of multi-camera fused projection correction.

Description

Technical Field

[0001] This application relates to the field of projection technology, specifically to an automatic correction method and system for fused projection. Background Technology

[0002] Projection calibration, especially geometric calibration in multi-projector fusion projection (i.e., multiple projectors working together to project onto larger areas), does indeed face many technical challenges. Therefore, in existing technologies, multi-projector fusion projection is mainly calibrated manually.

[0003] Traditional manual correction methods are not only time-consuming and labor-intensive, but also struggle to guarantee accuracy and consistency. While some automatic correction methods exist, these are often only suitable for single-camera systems, and their correction results are less than ideal in large fields of view or complex scenes. This is because image distortion is more complex in large fields of view, making it difficult for a single camera to capture comprehensive information, thus increasing the difficulty of correction. Furthermore, conventional multi-camera vision techniques suffer from difficulties in matching feature points (corresponding points). Due to differences in the angles and positions of different cameras, feature points in the images are difficult to accurately match, affecting the accuracy of correction. Simultaneously, the fusion algorithm itself is complex, requiring the handling of overlapping, stitching, and color adjustment issues between multiple images, further increasing the difficulty and cost of multi-camera fusion projection correction. Summary of the Invention

[0004] In view of the above problems, this application provides an automatic correction method for fused projection to solve the technical problems of low correction efficiency and accuracy of multi-machine fused projection.

[0005] To achieve the above objectives, this application provides an automatic correction method for blended projection, wherein the blended projection uses two or more projectors to project simultaneously, and the automatic correction method includes the following steps:

[0006] Model the receiving surface of the projection to obtain the receiving surface model;

[0007] Construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space;

[0008] Based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, the pixel correspondence between the receiving surface model and the projector space is obtained; the content to be projected is mapped to the projector space according to the pixel correspondence.

[0009] Furthermore, it includes multiple cameras, each camera corresponding to a different area of ​​the receiving surface;

[0010] The step of constructing the pixel mapping between the camera space and the projector space, and the pixel mapping between the camera space and the receiving surface model space, includes: constructing a pixel mapping between the first camera space and the projector space of the projector visible to the first camera; constructing a pixel mapping between the first camera and the receiving surface model space; the first camera is one of the multiple cameras; constructing a pixel mapping between the second camera and the projector space of the projector commonly visible to the first camera, obtaining a pixel mapping between the second camera and the first camera; constructing a pixel mapping between the third camera and the projector space of the projector commonly visible to the second camera and the second camera, obtaining a pixel mapping between the second camera and the third camera; based on the pixel mapping between the second camera and the first camera, and the pixel mapping between the second camera and the third camera, obtaining a pixel mapping between the third camera and the first camera; and based on the constructed pixel mappings, establishing pixel mappings between the remaining cameras and the first camera one by one, obtaining mutual mappings between each camera and the first camera.

[0011] The step of obtaining the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, includes: designating each projector as the current projector, identifying the camera that can capture the projection light path of the current projector and designating it as the current camera, projecting the pixels of the receiving surface model onto the first camera space, and then using the mutual mapping to project them onto the current camera space; and using the pixel mapping between the current projector and the current camera to project them onto the current projector space.

[0012] Furthermore, it includes one or more cameras, with at least one camera having multiple camera positions, and the camera corresponding to different areas of the receiving surface at different camera positions;

[0013] The construction of pixel mappings between the camera space and the projector space, and between the camera space and the receiving surface model space, includes: constructing a pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position; constructing a pixel mapping between the camera space at the first camera position and the receiving surface model space; moving the camera to a second camera position, and constructing a pixel mapping between the projector space of the projector commonly visible to both the second and first camera positions, thus obtaining a pixel mapping between the camera spaces at the second camera position and the first camera position. Shoot; move the camera to the third camera position, construct a pixel mapping of the projector space of the projector that is visible to both the third and second camera positions, and obtain a pixel mapping between the second and third camera positions; based on the pixel mapping between the second and first camera positions and the pixel mapping between the second and third camera positions, obtain a pixel mapping between the third and first camera positions; based on the constructed pixel mapping, establish pixel mappings between the camera space at the remaining camera positions and at the first camera position, and obtain mutual mappings between the camera space at each camera position and at the first camera position;

[0014] The step of obtaining the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, includes: designating each projector as the current projector; determining the camera position that can completely capture the projection light path of the current projector and designating it as the current camera position; projecting the pixels of the receiving surface model onto the camera space of the first camera position; and then using the mutual mapping to project them onto the camera space of the current camera position; and using the pixel mapping between the current projector and the current camera to project them onto the current projector space.

[0015] Furthermore, the pixel mapping between the camera space and the projector space is constructed based on the structured light sequence map generated by the projector, including:

[0016] The projector projects a structured light sequence image onto the receiving surface; the camera corresponding to the projector synchronously captures each structured light sequence image, and then applies a threshold to convert each captured structured light sequence image into a binary sequence image, and decodes the encoded value of each pixel in the binary sequence image; finds the pixel point in the projector space that has the same encoding as the pixel, and marks it as a pair of pixels, the pair of pixels being a pixel mapping between the projector space and the camera space.

[0017] Furthermore, it includes multiple cameras, each camera corresponding to a different area of ​​the receiving surface;

[0018] The step of constructing the pixel mapping between the camera space and the projector space, and the pixel mapping between the camera space and the receiving surface model space, includes: constructing a pixel mapping between the first camera space and the projector space of the projector visible to the first camera, denoted as pixel mapping one; constructing a pixel mapping between the first camera and the receiving surface model, denoted as pixel mapping two; the first camera is one of the plurality of cameras; constructing a pixel mapping between another camera and any of the projectors visible to the first camera, denoted as pixel mapping three; the other camera is another camera among the plurality of cameras besides the first camera;

[0019] The pixel mapping between the 2D coordinates of each pixel of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained based on pixel mapping one and pixel mapping two, and is denoted as pixel mapping four.

[0020] Based on pixel mapping three and pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, denoted as pixel mapping five.

[0021] Based on the point-to-point relationship of the pixel mapping five, an overall pixel mapping between the camera space of the other camera and the receiving surface model space is constructed, denoted as pixel mapping six;

[0022] According to the construction method of Mapping 3 to Mapping 6, construct the pixel mapping between the camera space of the remaining cameras and the receiving surface model space;

[0023] The step of obtaining the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping of the camera space, the projector space, and the receiving surface model space includes:

[0024] Each projector is designated as the current projector. A camera that can completely capture the projection light path of the current projector is designated as the current camera. The pixel mapping between the current camera and the receiving surface model space is used to project each pixel of the receiving surface model onto the current camera space.

[0025] Project onto the current projector space using the pixel mapping between the current camera and the projector.

[0026] Furthermore, the receiving surface model includes any one of a plane, a circular arc surface, or an irregular surface.

[0027] To solve the above-mentioned technical problems, this application provides another technical solution:

[0028] An automatic correction system for blended projection, wherein the blended projection uses two or more projectors to project simultaneously, the automatic correction system comprising:

[0029] The modeling module is used to model the bearing surface of the projection to obtain the bearing surface model;

[0030] A pixel mapping module is used to construct a pixel mapping between the camera space and the projector space, as well as a pixel mapping between the camera space and the receiving surface model space.

[0031] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space, the projector space, and the receiving surface model space; and to map the content to be projected onto the projector space according to the pixel correspondence.

[0032] Furthermore, the automatic correction system includes multiple cameras, each camera corresponding to a different area of ​​the bearing surface;

[0033] The pixel mapping module is used to construct pixel mappings between camera space and projector space, and between camera space and receiving surface model space. This includes: constructing a pixel mapping between a first camera space and the projector space of the projector visible to the first camera; constructing a pixel mapping between the first camera and the receiving surface model space; the first camera is one of a plurality of cameras; constructing a pixel mapping between a second camera and the projector space of the projector commonly visible to the first camera, obtaining a pixel mapping between the second camera and the first camera; constructing a pixel mapping between a third camera and the projector space commonly visible to the second camera and the second camera, obtaining a pixel mapping between the second camera and the third camera; based on the pixel mappings between the second camera and the first camera, and the second camera and the third camera, obtaining a pixel mapping between the third camera and the first camera; and based on the constructed pixel mappings, establishing pixel mappings between the remaining cameras and the first camera one by one, obtaining mutual mappings between each camera and the first camera.

[0034] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, and to map the content to be projected to the projector space according to the pixel correspondence, including: designating each projector as the current projector, identifying the camera that can capture the projection light path of the current projector and designating it as the current camera, projecting the pixels of the receiving surface model to the first camera space, and then using the mutual mapping to project to the current camera space; and using the pixel mapping between the current projector and the current camera to project to the current projector space.

[0035] Furthermore, the automatic correction system includes one or more cameras, and at least one camera is set with multiple camera positions. At different camera positions, the camera corresponds to different areas of the bearing surface.

[0036] The pixel mapping module is used to construct pixel mappings between the camera space and the projector space, and between the camera space and the receiving surface model space. This includes: constructing pixel mappings between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position, and pixel mappings between the camera space at the first camera position and the receiving surface model space; moving the camera to a second camera position, and constructing pixel mappings between the camera space and the visible projector space at the second camera position, thus obtaining pixel mappings between the camera space at the second camera position and the pixel mapping between the camera space at the first camera position and the pixel mapping between the camera space at the second camera position and the pixel mapping between the projector space at the first camera position. Pixel mapping; move the camera to the third camera position, construct a pixel mapping of the projector space of the projector that is visible to both the third and second camera positions, and obtain the pixel mapping between the second and third camera positions; based on the pixel mapping between the second and first camera positions and the pixel mapping between the second and third camera positions, obtain the pixel mapping between the third and first camera positions; based on the constructed pixel mapping, establish pixel mappings between the camera space at the remaining camera positions and at the first camera position one by one, and obtain the mutual mapping between the camera space at each camera position and at the first camera position;

[0037] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, and to map the content to be projected to the projector space according to the pixel correspondence, including: successively marking each projector as the current projector, determining the camera position that can completely capture the projection light path of the current projector and marking it as the current camera position, projecting the pixels of the receiving surface model to the camera space of the first camera position, and then using the mutual mapping to project to the camera space of the current camera position; and using the pixel mapping between the current projector and the current camera to project to the current projector space.

[0038] Furthermore, the automatic correction system includes multiple cameras, each corresponding to a different region of the bearing surface; the pixel mapping module is used to construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the bearing surface model space, including:

[0039] A pixel mapping is constructed between the space of the first camera and the projector space of the projector visible to the first camera, denoted as pixel mapping one; a pixel mapping is constructed between the first camera and the receiving surface model, denoted as pixel mapping two; the first camera is one of the multiple cameras.

[0040] Construct a pixel mapping between another camera and any of the projectors visible to the first camera, denoted as pixel mapping three; the other camera is another camera among the plurality of cameras besides the first camera;

[0041] The pixel mapping between the 2D coordinates of each pixel of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained based on pixel mapping one and pixel mapping two, and is denoted as pixel mapping four.

[0042] Based on pixel mapping three and pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, denoted as pixel mapping five.

[0043] Based on the point-to-point relationship of the pixel mapping five, an overall pixel mapping between the camera space of the other camera and the receiving surface model space is constructed, denoted as pixel mapping six;

[0044] Following the construction methods of Mappings Three to Six, construct pixel mappings between the camera spaces of the remaining cameras and the receiving surface model space.

[0045] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space, the projector space, and the receiving surface model space, and to map the content to be projected onto the projector space according to the pixel correspondence, including:

[0046] Each projector is designated as the current projector. A camera that can completely capture the projection light path of the current projector is designated as the current camera. The pixel mapping between the current camera and the receiving surface model space is used to project each pixel of the receiving surface model onto the current camera space.

[0047] Project onto the current projector space using the pixel mapping between the current camera and the projector.

[0048] Unlike existing technologies, the above-mentioned automatic correction method and system for fusion projection constructs a pixel mapping between the camera space and the projector space, as well as a pixel mapping between the camera space and the receiving surface model space, in fusion projection. Therefore, based on the pixel mappings of the camera space, the projector space, and the receiving surface model space, the pixel correspondence between the receiving surface model and the projector space can be obtained. Thus, the content to be projected can be mapped to the projector space according to the pixel correspondence, thereby achieving automatic correction from the projector to the projection receiving surface in multi-camera fusion projection, greatly improving the efficiency and accuracy of multi-camera fusion projection correction. Furthermore, the above solution uses multi-camera or time-division multiplexing technology, expanding the field of view for automatic correction and enabling correction of larger receiving surfaces. Moreover, when performing multi-camera correction, structured light projected by the projector is used to establish connection feature points between cameras, making the correction more accurate and reducing computational load.

[0049] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0050] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0051] In the accompanying drawings of the instruction manual:

[0052] Figure 1 A flowchart illustrating the automatic correction method for fused projection as described in a specific implementation;

[0053] Figure 2 This is a schematic diagram illustrating the automatic correction of the fused projection as described in a specific implementation.

[0054] Figure 3 A flowchart illustrating a method for automatic correction of multi-camera fusion projection in a specific implementation;

[0055] Figure 4 This is a schematic diagram illustrating automatic projection correction using multiple cameras as described in a specific implementation.

[0056] Figure 5 This is a schematic diagram illustrating the calibration of two cameras and two projectors using a building as the contact surface, as a specific implementation method.

[0057] Figure 6 A flowchart of a multi-camera fusion projection automatic correction method according to another embodiment;

[0058] Figure 7 This is a block diagram of the automatic correction system for fused projection described in a specific implementation.

[0059] The reference numerals used in the above figures are explained as follows:

[0060] 700. Automatic correction system for blended projection; 701. Modeling module; 702. Pixel mapping module; 703. Correction module. Detailed Implementation

[0061] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0062] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0063] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0064] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0065] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0066] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0067] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0068] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0069] Unless otherwise expressly specified or limited, the terms "installation," "connection," "linking," "fixing," and "setting," as used in the description of the embodiments of this application, should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two components or the interaction between two components. For those skilled in the art to which this application pertains, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0070] Please see Figure 1This embodiment provides an automatic correction method for blended projection. In this automatic correction method, blended projection refers to using two or more projectors to project simultaneously. Different projectors are responsible for projecting different areas of the receiving surface, thereby achieving a larger area projection. Taking dual-projector blended projection as an example, projector one can be used to project the left side of the receiving surface, and projector two can be used to project the right side of the receiving surface. The projections from the left and right sides of the receiving surface are then combined to form a complete large-area projection. In this embodiment, the automatic correction method for blended projection automatically corrects each projector in the blended projection, enabling each projector to correct the content to be projected based on its relative position to the receiving surface and the shape of the receiving surface.

[0071] like Figure 1 As shown, in this embodiment, the automatic correction method for the fused projection includes the following steps:

[0072] S101. Model the receiving surface of the projection to obtain the receiving surface model;

[0073] S102. Construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space;

[0074] S103. Based on the pixel mapping of the camera space, the projector space, and the receiving surface model space, obtain the pixel correspondence between the receiving surface model and the projector space; map the content to be projected onto the projector space according to the pixel correspondence.

[0075] In step S101, the receiving surface of the projection is modeled to obtain a receiving surface model including vertices and UV coordinates. The UV coordinates are a two-dimensional coordinate system, where U represents the horizontal direction and V represents the vertical direction. U and V together constitute a two-dimensional UV coordinate system.

[0076] In step S102, a pixel mapping between camera space and projector space can be constructed using structured light, and a pixel mapping between camera space and the receiving surface model space can be constructed using feature point matching. In this step, the structured light is generated by the projector and projected onto the receiving surface. The projector can project a sequence of structured light images, and the camera corresponding to the projector synchronously captures each sequence of structured light images. Then, a threshold is applied to convert each sequence of structured light images into a binary sequence image, and the encoded value of each pixel in the binary sequence image is decoded. Pixels with the same encoding as those in the projector space are found and marked as paired pixels, i.e., the pixel mapping between projector space and camera space. The projector projects a structured light pattern onto the receiving surface; the camera corresponding to the projector synchronously captures the structured light pattern, and then aligns and decodes it; according to the pixel encoding, the paired pixels in the camera space and projector space are found, and the paired pixels are the pixel mapping between projector space and camera space.

[0077] In step S103, the pixel coordinates of each pixel in the receiving surface model are projected into the camera space using the pixel mapping between the camera space and the receiving surface model space obtained in step S102; then, using the pixel mapping between the camera space and the projector space obtained in step S102, they are projected into the projector space. Finally, the content to be projected (e.g., video) is mapped into the projector space according to the UV coordinates and the pixel correspondence of the receiving surface model. It should be noted that in this embodiment, the pixels of the content to be projected are mapped into the receiving surface model according to the UV coordinates and the pixel correspondence of the receiving surface model; then, according to the pixel mapping between the camera space and the receiving surface model space, it is mapped into the camera space; and finally, according to the pixel mapping between the camera space and the projector space, it is mapped into the projector space.

[0078] This automatic correction method for fused projection is applicable to both single-camera and multi-camera applications. When a single camera can capture the entire projection surface, only one camera is needed to implement the automatic correction method for fused projection. Figure 2 As shown, it is a schematic diagram of automatic correction for fused projection of two projectors using a single camera.

[0079] When the field of view of a single camera cannot cover the entire projection surface, a multi-camera approach is used to perform the aforementioned automatic correction method for fused projection. Each camera is used to acquire images of different areas of the projection surface. For example... Figure 4 The diagram shown is a schematic of automatic correction for fused projection of dual projectors using two cameras. Figure 4 In the middle, the camera in the lower left corner (Camera 1) is used to capture the data from the receiving surface. Figure 4The image on the left side of the rectangle in the middle is captured by one camera (camera 2) in the lower right corner, and the image on the right side of the receiving surface is captured by another camera (camera 1) in the upper left corner. The image on the left side of the receiving surface is projected by one projector (projector 1) in the upper right corner, and the image on the right side of the receiving surface is projected by another projector (projector 2) in the upper right corner.

[0080] exist Figure 2 and Figure 4 In the diagram, the green arrow indicates the pixel mapping process of the projector in the upper left corner; the red arrow indicates the pixel mapping process of the projector in the upper right corner. These arrows only represent the process, not the order of operations. Figure 4 For the case of a dual-camera, dual-projector planar system, the operations for the two red lines on the left are the same as those for the green lines, and can be completed in the same step. The main difference between the automatic correction for fusion projection using a single camera and the automatic correction for fusion projection using multiple cameras lies in step S103. When using single-camera correction, it is only necessary to establish a pixel mapping between the camera and the receiving surface model; however, when using multi-camera correction, it is necessary to establish a pixel mapping between each camera and its corresponding region on the receiving surface model.

[0081] In this embodiment, based on the pixel mapping between the camera space, the projector space, and the receiving surface model space, the pixel correspondence between the receiving surface model and the projector space can be obtained. Therefore, the content to be projected can be mapped to the projector space according to the pixel correspondence, thereby realizing automatic correction from the projector to the projection receiving surface during multi-machine fusion projection, which greatly improves the efficiency and accuracy of multi-machine fusion projection correction.

[0082] like Figure 3 As shown, this embodiment provides an automatic correction method for fusion projection using multiple cameras. This automatic correction method for fusion projection includes multiple cameras, each corresponding to a different region of the receiving surface; the automatic correction method for fusion projection includes the following steps:

[0083] S301. Model the receiving surface of the projection to obtain the receiving surface model;

[0084] S302. Construct a pixel mapping between the first camera space and the projector space of the projector visible to the first camera, and construct a pixel mapping between the first camera and the receiving surface model space; the first camera is one of the multiple cameras.

[0085] S303. Construct a pixel mapping of the projector space of the projector that is visible to both the second camera and the first camera, to obtain a pixel mapping between the second camera and the first camera; construct a pixel mapping of the projector space of the projector that is visible to both the third camera and the second camera, to obtain a pixel mapping between the second camera and the third camera; based on the pixel mapping between the second camera and the first camera, and the pixel mapping between the second camera and the third camera, obtain a pixel mapping between the third camera and the first camera; based on the constructed pixel mappings, establish pixel mappings between the remaining cameras and the first camera one by one, to obtain mutual mappings between each camera and the first camera;

[0086] S304. Each of the projectors is recorded as the current projector. The camera that can capture the projection light path of the current projector is identified and recorded as the current camera. The pixels of the receiving surface model are projected to the first camera space, and then the mutual mapping is used to project them to the current camera space.

[0087] S305. Project the image onto the current projector space using the pixel mapping between the current projector and the current camera.

[0088] In this embodiment, the pixel mapping relationship between the receiving surface model and the projector is achieved by constructing camera inter-maps between multiple cameras. In step S302, a pixel mapping between the first camera space and the projector space can be constructed using structured light and labeled as pixel mapping a; a pixel mapping between the first camera and the receiving surface model space is constructed using feature point matching and labeled as pixel mapping b. In step S303, for the second camera, a pixel mapping between the second camera space and the space of a visible projector in the first camera is constructed using structured light and labeled as pixel mapping c; a pixel mapping between the second camera space and the first camera space is constructed using pixel mapping a and pixel mapping c. Similarly, if a third camera exists, a pixel mapping between the third camera space and the second camera can be constructed, and so on, constructing inter-maps between all cameras and the first camera. In step S303, the mapping between the second camera and the first camera is constructed using a projector. The inter-maps between other cameras and the first camera are further constructed based on the previously constructed pixel mappings. For example:

[0089] Construct mapping 1, which is the mapping between the first camera and a certain projector;

[0090] Construct mapping 2, which is the mapping between the second camera and the aforementioned projector;

[0091] The pixel mapping between the first camera and the second camera can be obtained through mapping 1 and mapping 2.

[0092] The pixel mapping between the third camera and the first camera is achieved through:

[0093] Construct mapping 3, which is a mapping between the second camera and a certain projector.

[0094] Construct mapping 4, which is a mapping between the third camera and a certain projector;

[0095] The pixel mapping between the third camera and the second camera can be obtained through mappings 3 and 4. Then, the mutual mapping between the third camera and the first camera can be obtained through the previously constructed pixel mapping between the first camera and the second camera.

[0096] In other words, except for the second camera which is directly mapped to the first camera, the mappings of the other cameras to the first camera are built up step by step based on the already constructed pixel mappings.

[0097] In steps S304 and S305, when any projector is designated as the current projector, a camera that can completely capture the projection light path of the current projector is identified and designated as the current camera. First, pixel mapping b is applied to project the pixels on the surface of the receiving surface model onto the first camera. Then, the mutual mapping between the first camera and the current camera is applied to project the image onto the current camera. Finally, the pixel mapping between the projector and the current camera is applied to project the image onto the projector.

[0098] In another embodiment, a single camera is moved to different positions multiple times over a period of time to replace the aforementioned automatic correction method using multiple cameras for fusion projection. In this automatic correction method, one or more cameras can be used, with at least one camera positioned at multiple locations. At different locations, each camera corresponds to a different area of ​​the receiving surface. For example, a camera may have three locations. At the first location, the camera captures the area on the left side of the receiving surface; at the second location, it captures the area in the middle of the receiving surface; and at the third location, it captures the area on the right side of the receiving surface. Therefore, by using three locations with a single camera, the receiving surface that could not be fully captured before can be covered.

[0099] The automatic correction method for fused projection, which employs multiple camera positions moved to different positions over time, includes:

[0100] Pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position; pixel mapping between the camera space at the first camera position and the receiving surface model space.

[0101] Move the camera to a second camera position and construct a pixel mapping of the projector space of the projector that is visible to both the second and first camera positions, thus obtaining a pixel mapping between the camera space at the second camera position and the first camera position. Move the camera to a third camera position and construct a pixel mapping of the projector space of the projector that is visible to both the third and second camera positions, thus obtaining a pixel mapping between the second and third camera positions. Based on the pixel mapping between the second and first camera positions and the pixel mapping between the second and third camera positions, obtain a pixel mapping between the third and first camera positions. Based on the constructed pixel mappings, establish pixel mappings between the remaining camera positions and the camera space at the first camera position one by one, thus obtaining mutual mappings between the camera space at each camera position and the camera space at the first camera position.

[0102] Each projector is recorded as the current projector. The camera position that can capture the projection light path of the current projector is determined and recorded as the current camera position. The pixels of the receiving surface model are projected onto the camera space of the first camera position, and then the mutual mapping is used to project onto the camera space of the current camera position.

[0103] The pixel mapping between the current projector and the current camera is used to project onto the current projector space.

[0104] In this embodiment, fewer cameras are needed to capture the entire bearing surface and complete automatic projection correction. In this embodiment, one camera can replace all cameras; that is, only one camera is required. For example, where three cameras were originally used for automatic correction, only one camera with three camera positions is used, each with a shooting range corresponding to the original camera. In other embodiments, one or two cameras can replace some of the cameras in the above embodiments. For example, where three cameras were originally used for automatic correction, one camera (with two different camera positions) can replace two of the cameras, while retaining the third camera. As another example, where four cameras were originally used for automatic correction, two cameras can be used instead, each with two camera positions, replacing the original two cameras.

[0105] like Figure 6 As shown, another embodiment illustrates an automatic correction method for fused projection using multiple cameras. In this embodiment, multiple cameras are included, each corresponding to a different region of the receiving surface, and multi-camera correction is achieved by constructing a pixel mapping between each camera and the receiving surface model.

[0106] like Figure 6 As shown, the automatic correction method for multi-camera fusion projection in this embodiment includes:

[0107] S601. Construct a pixel mapping between the first camera space and the visible projector using structured light, denoted as pixel mapping one;

[0108] S602. Construct a pixel mapping between the first camera space and the receiving surface space through feature point matching, denoted as pixel mapping two;

[0109] S603. For the second camera, a visible projector in the first camera is visible. This projector is denoted as Projector 1. A pixel mapping between the space of the second camera and the space of Projector 1 is constructed by structured light and denoted as Pixel Mapping 3. Here, "the second camera can see a visible projector in the first camera" means that the second camera can capture a part of the light path of the projector captured by the first camera. For example, if the first camera can capture Projector 1, the second camera can capture a part of the light path of Projector 1.

[0110] S604. Based on pixel mapping one and pixel mapping two, the pixel mapping between the 2D coordinates of each pixel of the projector and the 3D coordinates of each point projected onto the receiving surface model can be obtained, which is denoted as pixel mapping four.

[0111] S605. Based on pixel mapping three and pixel mapping four, a partial pixel mapping of the 3D points of the second camera and the receiving surface model can be obtained, denoted as pixel mapping five.

[0112] S606. Based on the point-to-point relationship of pixel mapping five, construct the overall pixel mapping between the second camera space and the receiving surface model space, denoted as pixel mapping six;

[0113] S607. Following steps S603-S606, calculate the pixel mapping between the remaining cameras and the receiving surface model space.

[0114] S608. For any projector, find the camera that can completely capture the projection light path of the projector, and designate it as the current camera. Apply the pixel mapping between the current camera and the receiving surface model space to project the pixels of the receiving surface model onto the current camera space; then apply the pixel mapping between the current camera space and the projector space to project onto the projector. Through step S608, each projector can be designated as the current projector, the camera that can completely capture the projection light path of the current projector can be identified and designated as the current camera, the pixel mapping between the current camera and the receiving surface model space can be used to project each pixel of the receiving surface model onto the current camera space, and the pixel mapping between the current camera and the projector can be used to project onto the current projector space.

[0115] like Figure 2 , Figure 4 and Figure 5 As shown, in the above embodiments, the receiving surface of the projection can be any one of a plane, a curved surface, or an irregular surface. For example... Figure 2The bearing surface shown is a cylindrical surface (i.e., a type of circular arc surface). Figure 4 The bearing surface shown is a rectangular bearing surface (i.e., one type of plane). Figure 5 The surface shown is the building facade (i.e., one type of irregular surface).

[0116] like Figure 2 The diagram shown illustrates automatic correction of fused projection using a single camera and two projectors on a cylindrical surface. This method for automatic correction of fused projection on a cylindrical surface includes the following steps:

[0117] 11. Position the camera so that it can capture the entire cylindrical surface (i.e., the surface on which the projection is received).

[0118] 12. Modeling the cylinder: Measure the shape parameters of the cylinder and model it. When modeling, use the upper left corner of the cylinder as (0,0) and the lower right corner as (1,1) to obtain the UV coordinates.

[0119] 13. Calculate the pixel mapping between camera space and projector space: In this embodiment, a binary structured light algorithm is selected to construct the pixel mapping between the camera and the projector. The first projector projects a sequence of structured light images, and the camera synchronously captures each sequence. A threshold is applied to convert each sequence into a binary image. The encoding of the projected pixel that matches the binary sequence of each pixel is found and marked as a pair of pixels, i.e., the pixel mapping between the first projector space and the camera space. Similarly, the pixel mapping between the second projector space and the camera space can be calculated.

[0120] 14. Calculate the pixel mapping between camera space and the receiving surface (cylinder) space: The camera captures images of the cylinder, and an edge detection algorithm is applied to extract the cylinder's edge points. These edge points are then fitted with a Bézier surface to represent the image of the cylinder projected onto the camera's imaging surface.

[0121] 15. Projecting cylindrical pixels into projector space: For any cylindrical pixel, its UV coordinates can be found. Substituting the UV coordinates into the Bézier surface formula, we can obtain its 2D coordinates projected into the camera space. Combining the pixel mapping between the camera space and the projector space in step 13, we can find the corresponding projector, i.e., the 2D coordinates of the projector space.

[0122] 16. Video UV Mapping to Projector Space: Acquire a video, which is a planar video created by unfolding cylindrical UV coordinates. For any pixel in the video, substitute its UV coordinates into the model in step 12 to obtain the model's pixel coordinates, and substitute these into step 15 to project it onto the projector spaces. Projecting all video pixels yields the corrected image.

[0123] like Figure 4The diagram shown illustrates the automatic correction of fused projection using a dual-camera, dual-projector system with a planar receiving surface. The automatic correction method for fused projection using this dual-camera, dual-projector system includes the following steps:

[0124] 21. Position the cameras so that camera one captures the left side of the receiving surface and camera two captures the right side of the receiving surface; position the projector so that projector one projects onto the left side of the receiving surface and projector two projects onto the right side of the receiving surface. Adjust camera two so that it can capture a portion of the light path from projector one.

[0125] 22. Modeling: Measure the width and height of the bearing surface, construct the model of the bearing surface, and expand it according to the upper left corner (0,0) and the lower right corner (1,1) of the bearing surface to obtain the UV coordinates.

[0126] 23. Calculate the pixel mapping between camera space and projector space: Projector 1 projects a structured light sequence image, which is simultaneously captured and recognized by Camera 1 and Camera 2, resulting in the pixel mapping A between projector space and camera space, and the pixel mapping B between projection space and camera space. Substitute the point pairs of pixel mapping A into the least squares method to solve for the homography matrix A between projection space 1 and camera space 1; similarly, the homography matrix B between projection space 1 and camera space 2 is obtained. Projector 2 projects a structured light sequence image, which is captured and recognized by camera 2, and the homography matrix is ​​solved to obtain the homography matrix C between projection space 2 and camera space 2.

[0127] 24. Calculate the pixel mapping between camera space and receiving surface (plane) space: Mark 4 points on the receiving surface and obtain the 4 points corresponding to the marked points in camera one. Solve the homography matrix D from these 4 point pairs, that is, the pixel mapping between camera one space and receiving surface space.

[0128] 25. Projecting the pixels of the receiving surface onto the projector (pixel mapping C): Obtain the pixels of the receiving surface, apply the homography matrix D to map them to the camera space, and then apply the homography matrix A to map them to the projection space.

[0129] 26. Projecting planar pixels onto projector two (pixel mapping D): Obtain the pixels of the receiving surface, apply homography matrix D, and map them to camera one space; then apply homography matrix A, and map them to projection one space; then apply homography matrix B, and map them to camera two space; then apply homography matrix C, and map them to projection two space.

[0130] 27. Video Mapping to Projector Space: Acquire a video, which is created using planar UV unwrapping. For any pixel, look up its corresponding receiving pixel using UV coordinates, and project it onto the corresponding projector space using pixel mapping C or pixel mapping D to obtain the corrected image.

[0131] like Figure 5The diagram illustrates the automatic correction of blended projection using a dual-camera, dual-projector system with a building as the receiving surface. In the diagram, "a" represents pixel mapping a; "b" represents pixel mapping b; "c" represents pixel mapping c; "d" represents pixel mapping d; "e" represents pixel mapping e; "f" represents pixel mapping f; and "g" represents pixel mapping g. The four circles represent four easily identifiable vertices on the receiving surface. This automatic correction method for blended projection using a dual-camera, dual-projector system with a building as the receiving surface includes the following steps:

[0132] 31. Using Zhang Zhengyou's calibration method, calibrate the intrinsic parameters and distortion coefficients of camera one and camera two.

[0133] 32. Position cameras so that camera one films the left side of the building and camera two films the right side; position projectors so that projector one onto the left side of the building and projector two onto the right side. Adjust camera two so that it can capture part of the light path of projector one.

[0134] 33. Bonding Surface Modeling: Using laser scanning, construct the bonding surface model and correctly unfold it in 3D software to obtain UV coordinates.

[0135] 34. Calculate the pixel mapping between the camera and the projector: Projector 1 projects a sequence of structured light images, which are simultaneously captured and recognized by Camera 1 and Camera 2, yielding the pixel mapping 'a' between the space of Projector 1 and the space of Camera 1, and the pixel mapping 'b' between the space of Projector 1 and the space of Camera 2. Projector 2 projects a sequence of structured light images, which are captured and recognized by Camera 2, yielding the pixel mapping 'c' between the space of Projector 2 and the space of Camera 2.

[0136] 35. Calculate the pixel mapping d between camera 1 and the receiving surface (building): On the left side of the building, find four easily identifiable vertices (e.g., Figure 5 The coordinates of the four circles in the image are obtained within camera one, resulting in four point pairs. Given the intrinsic parameters of camera one, the PnP algorithm can be applied to solve for the pose (spatial coordinates, yaw, pitch, and roll) of camera one relative to the building. Knowing the pose, intrinsic parameters, and distortion coefficients, the perspective matrix can be used to calculate the coordinates of each vertex of the model at the image point on camera one, thus obtaining the pixel mapping between camera one space and the receiving surface model.

[0137] 36. Calculate the pixel mapping e between camera 2 and the receiving surface: Combining pixel mapping a and pixel mapping d, the coordinates of each pixel in projection 1 on the receiving surface can be obtained; then, combining pixel mapping b, the pixel mapping relationship between some pixels of camera 2 and vertices of the receiving surface can be obtained, that is, the point-to-point relationship between several points in the space of camera 2 and several points in the model space of the receiving surface. Combining the intrinsic parameters of camera 2, the pose of camera 2 relative to the building can be solved by applying the PnP algorithm.

[0138] 37. Projecting building vertices onto projector 1 (pixel mapping f): Obtain the building vertices, apply pixel mapping d to get their coordinates on camera 1; then apply pixel mapping a to get their coordinates on projector 1.

[0139] 38. Projecting building vertices onto projector 2 (pixel mapping g): Obtain the building vertices, apply pixel mapping e to get their coordinates on camera 2; then apply pixel mapping c to get their coordinates on projector 2.

[0140] 39. Mapping content to the projector space. This includes two methods. Method 1: Find the vertex coordinates of the model for each pixel of the video using UV mapping, and then apply pixel mapping f or pixel mapping g to project it onto the projector.

[0141] Method 2 includes:

[0142] Step 391: Determine the projector pose: The projector can be mathematically represented as a pinhole camera model, therefore its intrinsic parameters can also be expressed as focal length and optical center. From pixel mappings f and g, point pairs between the projector and the model can be extracted. To solve the projector's pose relative to the building using the PnP algorithm, its intrinsic parameters need to be solved. A traversal search method can be used to solve the intrinsic parameters, which includes the following steps: treating the projector's focal length and optical center intrinsic parameters as dimensions of a multi-dimensional solution space, traversing and trying every combination in the solution space, applying the PnP algorithm to solve the projector pose under a specific combination, and calculating the reprojection coordinates of the model vertices under this pose to evaluate the reprojection error. After a complete traversal, the dimension combination with the minimum error is selected as the "equivalent intrinsic parameter".

[0143] Step 392: After obtaining the intrinsic and extrinsic parameters of the projector, the coordinates of each vertex of the model mapped to the projector can be calculated using the perspective matrix. Method 2, compared to Method 1, overcomes the problem of poor local correction caused by individual point recognition errors in structured light.

[0144] In the above embodiments, the camera space, projector space, receiving surface space or receiving surface model space all refer to the mathematical space under its coordinate system; 2) Sometimes the word "space" is omitted in the description, such as "projected to the first camera" actually means "projected to the first camera space".

[0145] like Figure 7 As shown, another embodiment provides an automatic correction system 700 for blended projection. The blended projection uses two or more projectors to project simultaneously. The automatic correction system 700 includes: a modeling module 701, a pixel mapping module 702, and a correction module 703.

[0146] Modeling module 701 is used to model the bearing surface of the projection to obtain the bearing surface model;

[0147] The pixel mapping module 702 is used to construct the pixel mapping between the camera space and the projector space, and the pixel mapping between the camera space and the receiving surface model space;

[0148] The correction module 703 is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space; and to map the content to be projected to the projector space according to the pixel correspondence.

[0149] In this embodiment, the receiving surface model includes any one of a plane, a circular arc surface, or an irregular surface.

[0150] The automatic correction system 700 for this fusion projection is applicable to both single-camera and multi-camera applications. When a single camera can capture the entire projection surface, only one camera is needed to implement the aforementioned automatic correction method for fusion projection. However, when the field of view of a single camera cannot cover the entire projection surface, multiple cameras are used in conjunction to perform the aforementioned automatic correction method for fusion projection. Each camera is used to acquire images of different areas of the projection surface.

[0151] In one embodiment, the automatic correction system includes a plurality of cameras, each camera corresponding to a different region of the bearing surface;

[0152] The pixel mapping module is used to construct pixel mappings between the camera space and the projector space, and between the camera space and the receiving surface model space, including:

[0153] Construct a pixel mapping between the first camera space and the projector space of the projector visible to the first camera, and construct a pixel mapping between the first camera and the receiving surface model space; the first camera is one of the plurality of cameras;

[0154] 1. Construct a pixel mapping of the projector space of the projector that is visible to both the second camera and the first camera, to obtain the pixel mapping between the second camera and the first camera; 2. Construct a pixel mapping of the projector space of the projector that is visible to both the third camera and the second camera, to obtain the pixel mapping between the second camera and the third camera; 3. Based on the pixel mapping between the second camera and the first camera, and the pixel mapping between the second camera and the third camera, obtain the pixel mapping between the third camera and the first camera; 4. Based on the constructed pixel mapping, establish the pixel mapping between the remaining cameras and the first camera one by one, to obtain the mutual mapping between each camera and the first camera;

[0155] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, and to map the content to be projected onto the projector space according to the pixel correspondence, including:

[0156] Each projector is designated as the current projector. A camera capable of capturing the projection light path of the current projector is designated as the current camera. The pixels of the receiving surface model are projected onto the first camera space, and then the mutual mapping is used to project them onto the current camera space.

[0157] The pixel mapping between the current projector and the current camera is used to project onto the current projector space.

[0158] In this embodiment, automatic correction of multi-camera fusion projection is achieved by constructing mutual mapping between cameras.

[0159] In another embodiment, the automatic correction system includes one or more cameras, at least one of the cameras being set at multiple camera positions, and at different camera positions, the camera corresponds to different areas of the bearing surface;

[0160] The pixel mapping module is used to construct pixel mappings between the camera space and the projector space, and between the camera space and the receiving surface model space, including:

[0161] The pixel mapping between the camera space when constructing the first camera position and the projector space of the projector visible to the camera at the current camera position; the pixel mapping between the camera space and the receiving surface model space when constructing the first camera position.

[0162] Move the camera to a second camera position and construct a pixel mapping of the projector space of the projector that is visible to both the second and first camera positions, thus obtaining a pixel mapping between the camera space at the second camera position and the first camera position. Move the camera to a third camera position and construct a pixel mapping of the projector space of the projector that is visible to both the third and second camera positions, thus obtaining a pixel mapping between the second and third camera positions. Based on the pixel mapping between the second and first camera positions and the pixel mapping between the second and third camera positions, obtain a pixel mapping between the third and first camera positions. Based on the constructed pixel mappings, establish pixel mappings between the remaining camera positions and the camera space at the first camera position one by one, thus obtaining mutual mappings between the camera space at each camera position and the camera space at the first camera position.

[0163] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, and to map the content to be projected onto the projector space according to the pixel correspondence, including:

[0164] Each projector is recorded as the current projector. The camera position that can completely capture the projection light path of the current projector is determined and recorded as the current camera position. The pixels of the receiving surface model are projected onto the camera space of the first camera position, and then the mutual mapping is used to project onto the camera space of the current camera position.

[0165] The pixel mapping between the current projector and the current camera is used to project onto the current projector space.

[0166] In this embodiment, a single camera is moved to different positions multiple times over a period of time to replace the aforementioned multiple cameras for automatic correction of the blended projection. In this embodiment, fewer cameras can be used to capture the entire bearing surface and complete the automatic correction of the blended projection.

[0167] In another embodiment, automatic correction of the multi-camera fusion projection can be achieved by constructing a pixel mapping between each camera space and the receiving surface space. The automatic correction system includes multiple cameras, each corresponding to a different region of the receiving surface.

[0168] The pixel mapping module is used to construct pixel mappings between the camera space and the projector space, and between the camera space and the receiving surface model space, including:

[0169] A pixel mapping is constructed between the space of the first camera and the projector space of the projector visible to the first camera, denoted as pixel mapping one; a pixel mapping is constructed between the first camera and the receiving surface model, denoted as pixel mapping two; the first camera is one of the multiple cameras.

[0170] Construct a pixel mapping between another camera and any of the projectors visible to the first camera, denoted as pixel mapping three; the other camera is another camera among the plurality of cameras besides the first camera;

[0171] The pixel mapping between the 2D coordinates of each pixel of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained based on pixel mapping one and pixel mapping two, and is denoted as pixel mapping four.

[0172] Based on pixel mapping three and pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, denoted as pixel mapping five.

[0173] Based on the point-to-point relationship of the pixel mapping five, an overall pixel mapping between the camera space of the other camera and the receiving surface model space is constructed, denoted as pixel mapping six;

[0174] Based on pixel mapping three to pixel mapping six, pixel mappings between each camera space and the receiving surface model space are obtained;

[0175] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, and to map the content to be projected onto the projector space according to the pixel correspondence, including:

[0176] Each projector is designated as the current projector. A camera that can completely capture the projection light path of the current projector is designated as the current camera. The pixel mapping between the current camera and the receiving surface model space is used to project each pixel of the receiving surface model onto the current camera space.

[0177] Project onto the current projector space using the pixel mapping between the current camera and the projector.

[0178] In the above embodiments, using multiple cameras for automatic correction of fused projection expands the field of view of the cameras during automatic correction, thereby enabling the correction of larger surfaces. Furthermore, during automatic correction of fused projection using multiple cameras, structured light projected by the projector is used to establish connection feature points between cameras, making the correction more accurate and reducing computational load. In some embodiments, a single camera can be moved to different positions in a time-sharing manner to replace multiple cameras, thereby reducing the number of cameras used and further improving applicability. Moreover, the above embodiments do not limit the type of surface, supporting tasks such as simple planar mapping and complex 3D mapping.

[0179] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. An automatic correction method for fused projection, characterized in that, The fused projection uses two or more projectors to project simultaneously, and the automatic correction method includes the following steps: Model the receiving surface of the projection to obtain the receiving surface model; Construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space; Based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, the pixel correspondence between the receiving surface model and the projector space is obtained; the content to be projected is mapped to the projector space according to the pixel correspondence. The construction of pixel mappings between the camera space and the projector space, and between the camera space and the receiving surface model space, includes one of the following methods: Method 1: Set up multiple cameras, each camera corresponding to a different area of ​​the receiving surface; Construct a pixel mapping between the first camera space and the projector space of the projector visible to the first camera, and construct a pixel mapping between the first camera and the receiving surface model space; the first camera is one of the plurality of cameras; Based on the pixel mapping of the projector space of the projector that is visible to both cameras, the mutual mapping between each camera and the first camera is obtained. Method 2: One or more cameras are set up, and at least one camera is set up with multiple camera positions. When the camera is in different positions, the camera corresponds to different areas of the receiving surface. Pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position; pixel mapping between the camera space at the first camera position and the receiving surface model space. Based on the pixel mapping of the projector space of the projector that is visible to both camera positions, the mutual mapping between the camera space at each camera position and the camera space at the first camera position is obtained. Method 3: Multiple cameras are set up, each camera corresponding to a different area of ​​the bearing surface; A pixel mapping is constructed between the space of the first camera and the projector space of the projector visible to the first camera, denoted as pixel mapping one; a pixel mapping is constructed between the first camera and the receiving surface model, denoted as pixel mapping two; the first camera is one of the multiple cameras. Based on constructing a pixel mapping between another camera and any of the projectors visible to the first camera, a pixel mapping between the camera space of the remaining cameras and the receiving surface model space is obtained.

2. The automatic correction method for fused projection according to claim 1, characterized in that, In the first method, obtaining the mutual mapping between each camera and the first camera based on the pixel mapping of the projector space of the projector, which is visible to both cameras, includes: Build a pixel mapping of the projector space of the projector that is visible to both the second camera and the first camera to obtain a pixel mapping between the second camera and the first camera; build a pixel mapping of the projector space of the projector that is visible to both the third camera and the second camera to obtain a pixel mapping between the second camera and the third camera; based on the pixel mapping between the second camera and the first camera, and the pixel mapping between the second camera and the third camera, obtain a pixel mapping between the third camera and the first camera. Based on the constructed pixel mapping, the pixel mapping between the remaining cameras and the first camera is established one by one to obtain the mutual mapping between each camera and the first camera; The step of obtaining the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, includes: Each projector is designated as the current projector. A camera capable of capturing the projection light path of the current projector is designated as the current camera. The pixels of the receiving surface model are projected onto the first camera space, and then the mutual mapping is used to project them onto the current camera space. The pixel mapping between the current projector and the current camera is used to project onto the current projector space.

3. The automatic correction method for fused projection according to claim 1, characterized in that, In the second method, the pixel mapping of the projector space based on the projector visible to both camera positions, to obtain the mutual mapping between the camera space at each camera position and at the first camera position, includes: Move the camera to a second camera position and construct a pixel mapping of the projector space of the projector, which is visible to both the second and first camera positions, to obtain a pixel mapping between the camera space at the second camera position and the first camera position. Move the camera to a third camera position and construct a pixel mapping of the projector space of the projector, which is visible to both the third and second camera positions, to obtain a pixel mapping between the second and third camera positions. Based on the pixel mapping between the second and first camera positions and the pixel mapping between the second and third camera positions, obtain a pixel mapping between the third camera position and the first camera position. Based on the constructed pixel mapping, establish pixel mappings between the remaining camera positions and the camera space at the first camera position one by one to obtain mutual mappings between the camera space at each camera position and the camera space at the first camera position. The step of obtaining the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, includes: Each projector is recorded as the current projector. The camera position that can completely capture the projection light path of the current projector is determined and recorded as the current camera position. The pixels of the receiving surface model are projected onto the camera space of the first camera position, and then the mutual mapping is used to project onto the camera space of the current camera position. The pixel mapping between the current projector and the current camera position is used to project onto the current projector space.

4. The automatic correction method for fused projection according to claim 2 or 3, characterized in that, The pixel mapping between the camera space and the projector space is constructed based on the structured light sequence map generated by the projector, including: The projector projects a structured light sequence diagram onto the receiving surface; The projector's corresponding camera synchronously captures each structured light sequence image, then applies a threshold to convert each captured structured light sequence image into a binary sequence image, and decodes the encoded value of each pixel in the binary sequence image; Find the pixels in the projector space that are the same as the encoded pixels, and mark them as paired pixels. The paired pixels are the pixel mapping between the projector space and the camera space.

5. The automatic correction method for fused projection according to claim 1, characterized in that, In the third method, obtaining the pixel mapping between the camera space of the remaining cameras and the receiving surface model space based on constructing a pixel mapping between another camera and any of the projectors visible to the first camera includes: Construct a pixel mapping between another camera and any of the projectors visible to the first camera, denoted as pixel mapping three; the other camera is another camera among the plurality of cameras besides the first camera; The pixel mapping between the 2D coordinates of each pixel of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained based on pixel mapping one and pixel mapping two, and is denoted as pixel mapping four. Based on pixel mapping three and pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, denoted as pixel mapping five. Based on the point-to-point relationship of the pixel mapping five, an overall pixel mapping between the camera space of the other camera and the receiving surface model space is constructed, denoted as pixel mapping six; According to the construction method of Mapping 3 to Mapping 6, construct the pixel mapping between the camera space of the remaining cameras and the receiving surface model space; The step of obtaining the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping of the camera space, the projector space, and the receiving surface model space includes: Each projector is designated as the current projector. A camera that can completely capture the projection light path of the current projector is designated as the current camera. The pixel mapping between the current camera and the receiving surface model space is used to project each pixel of the receiving surface model onto the current camera space. Project onto the current projector space using the pixel mapping between the current camera and the projector.

6. The automatic correction method for fused projection according to claim 1, characterized in that, The receiving surface model includes any one of the following: a plane, a circular arc, or an irregular surface.

7. An automatic correction system for fused projection, characterized in that, The fused projection uses two or more projectors to project simultaneously, and the automatic correction system includes: The modeling module is used to model the bearing surface of the projection to obtain the bearing surface model; A pixel mapping module is used to construct a pixel mapping between the camera space and the projector space, as well as a pixel mapping between the camera space and the receiving surface model space. The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space; and to map the content to be projected onto the projector space according to the pixel correspondence. The construction of pixel mappings between the camera space and the projector space, and between the camera space and the receiving surface model space, includes one of the following methods: Method 1: Set up multiple cameras, each camera corresponding to a different area of ​​the receiving surface; Construct a pixel mapping between the first camera space and the projector space of the projector visible to the first camera, and construct a pixel mapping between the first camera and the receiving surface model space; the first camera is one of the plurality of cameras; Based on the pixel mapping of the projector space of the projector that is visible to both cameras, the mutual mapping between each camera and the first camera is obtained. Method 2: One or more cameras are set up, and at least one camera is set up with multiple camera positions. When the camera is in different positions, the camera corresponds to different areas of the receiving surface. Pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position; pixel mapping between the camera space at the first camera position and the receiving surface model space. Based on the pixel mapping of the projector space of the projector that is visible to both camera positions, the mutual mapping between the camera space at each camera position and the camera space at the first camera position is obtained. Method 3: Multiple cameras are set up, each camera corresponding to a different area of ​​the bearing surface; A pixel mapping is constructed between the space of the first camera and the projector space of the projector visible to the first camera, denoted as pixel mapping one; a pixel mapping is constructed between the first camera and the receiving surface model, denoted as pixel mapping two; the first camera is one of the multiple cameras. Based on constructing a pixel mapping between another camera and any of the projectors visible to the first camera, a pixel mapping between the camera space of the remaining cameras and the receiving surface model space is obtained.

8. The automatic correction system for fused projection according to claim 7, characterized in that, In the first method, obtaining the mutual mapping between each camera and the first camera based on the pixel mapping of the projector space of the projector, which is visible to both cameras, includes:

1. Construct a pixel mapping of the projector space of the projector that is visible to both the second camera and the first camera, to obtain the pixel mapping between the second camera and the first camera; 2. Construct a pixel mapping of the projector space of the projector that is visible to both the third camera and the second camera, to obtain the pixel mapping between the second camera and the third camera; 3. Based on the pixel mapping between the second camera and the first camera, and the pixel mapping between the second camera and the third camera, obtain the pixel mapping between the third camera and the first camera; 4. Based on the constructed pixel mapping, establish the pixel mapping between the remaining cameras and the first camera one by one, to obtain the mutual mapping between each camera and the first camera; The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, and to map the content to be projected onto the projector space according to the pixel correspondence, including: Each projector is designated as the current projector. A camera capable of capturing the projection light path of the current projector is designated as the current camera. The pixels of the receiving surface model are projected onto the first camera space, and then the mutual mapping is used to project them onto the current camera space. The pixel mapping between the current projector and the current camera is used to project onto the current projector space.

9. The automatic correction system for fused projection according to claim 7, characterized in that, In the second method, the step of obtaining the mutual mapping between the camera space at each camera position and the camera space at the first camera position based on the pixel mapping of the projector space of the projector commonly visible to both camera positions includes: moving the camera to the second camera position and constructing a pixel mapping of the projector space of the projector commonly visible to both the second and first camera positions to obtain a pixel mapping between the camera space at the second camera position and the first camera position; moving the camera to the third camera position and constructing a pixel mapping of the projector space of the projector commonly visible to both the third and second camera positions to obtain a pixel mapping between the second and third camera positions; obtaining a pixel mapping between the third camera position and the first camera position based on the pixel mapping between the second and first camera positions and the pixel mapping between the second and third camera positions; and establishing pixel mappings between the remaining camera positions and the camera space at the first camera position based on the constructed pixel mappings to obtain a mutual mapping between the camera space at each camera position and the camera space at the first camera position. The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space and the projector space, and between the camera space and the receiving surface model space, and to map the content to be projected onto the projector space according to the pixel correspondence, including: Each projector is recorded as the current projector. The camera position that can completely capture the projection light path of the current projector is determined and recorded as the current camera position. The pixels of the receiving surface model are projected onto the camera space of the first camera position, and then the mutual mapping is used to project onto the camera space of the current camera position. The pixel mapping between the current projector and the current camera position is used to project onto the current projector space.

10. The automatic correction system for fused projection according to claim 7, characterized in that, In the third method, obtaining the pixel mapping between the camera space of the remaining cameras and the receiving surface model space based on constructing a pixel mapping between another camera and any of the projectors visible to the first camera includes: Construct a pixel mapping between another camera and any of the projectors visible to the first camera, denoted as pixel mapping three; the other camera is another camera among the plurality of cameras besides the first camera; The pixel mapping between the 2D coordinates of each pixel of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained based on pixel mapping one and pixel mapping two, and is denoted as pixel mapping four. Based on pixel mapping three and pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, denoted as pixel mapping five. Based on the point-to-point relationship of the pixel mapping five, an overall pixel mapping between the camera space of the other camera and the receiving surface model space is constructed, denoted as pixel mapping six; According to the construction method of Mapping 3 to Mapping 6, construct the pixel mapping between the camera space of the remaining cameras and the receiving surface model space; The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space based on the pixel mapping between the camera space, the projector space, and the receiving surface model space, and to map the content to be projected onto the projector space according to the pixel correspondence, including: Each projector is designated as the current projector. A camera that can completely capture the projection light path of the current projector is designated as the current camera. The pixel mapping between the current camera and the receiving surface model space is used to project each pixel of the receiving surface model onto the current camera space. Project onto the current projector space using the pixel mapping between the current camera and the projector.

Images