Automatic correction method and system for fusion projection

Abstract

The invention discloses an automatic correction method and system for fusion projection, and the method comprises the steps: carrying out the modeling of a bearing surface of projection, and obtaining a bearing surface model; constructing pixel mapping between a camera space and the projector space and pixel mapping between the camera space and the bearing surface model space; according to pixel mapping of the camera space, the projector space and the bearing surface model space, obtaining a pixel corresponding relation between the bearing surface model and the projector space; and mapping the content to be projected to the projector space according to the pixel corresponding relation. According to pixel mapping of a camera space, a projector space and a bearing surface model space, a pixel corresponding relation between a bearing surface model and the projector space can be obtained; therefore, the content to be projected can be mapped to the projector space according to the pixel corresponding relation, automatic correction during multi-machine fusion projection is realized, and the efficiency and precision of multi-machine fusion projection correction are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of projection technology, and in particular to an automatic correction method and system for fusion projection. Background Art

[0002] Projection correction, especially geometric correction in multi-machine fusion projection (i.e., multiple projectors collaborate to perform larger-area projection), does face many technical difficulties. Therefore, in the prior art, multi-machine fusion projection is mainly corrected manually.

[0003] Traditional manual correction methods are not only time-consuming and labor-intensive, but also difficult to ensure the accuracy and consistency of the correction. Although there are some automatic correction methods on the market, these methods are often only applicable to single-camera systems, and the correction effect is not ideal in large fields of view or complex scenes. This is because the image deformation under a large field of view is more complicated, and it is difficult for a single camera to capture comprehensive information, which makes correction more difficult. In addition, conventional multi-camera vision technology has the problem of difficulty in matching feature points (same-name points). Due to the differences in shooting angles and positions of different cameras, it is difficult for the feature points in the image to accurately correspond, which in turn affects the accuracy of the correction. At the same time, the fusion algorithm itself is also complex and needs to deal with issues such as overlap, splicing, and color adjustment between multiple images, which also increases the difficulty and cost of multi-machine fusion projection correction. Summary of the invention

[0004] In view of the above problems, the present application provides an automatic correction method for fused projection, which is used to solve the technical problems of low efficiency and accuracy of multi-machine fused projection correction.

[0005] To achieve the above object, the present application provides an automatic correction method for fused projection, wherein the fused projection is projected by two or more projectors, and the automatic correction method comprises the following steps:

[0006] Modeling the receiving surface of the projection to obtain a receiving surface model;

[0007] Constructing a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space;

[0008] According to the pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space, the pixel correspondence between the receiving surface model and the projector space is obtained; and the content to be projected is mapped to the projector space according to the pixel correspondence.

[0009] Furthermore, it comprises a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface;

[0010] The constructing of the pixel mapping between the camera space and the projector space, and the pixel mapping between the camera space and the interface model space, includes: constructing a pixel mapping between the first camera space and the projector space of the projector visible to the first camera, and constructing a pixel mapping between the first camera and the interface model space; the first camera is one of the multiple cameras; constructing a pixel mapping of the projector space of the projector visible to the second camera and the first camera, and obtaining a pixel mapping between the second camera and the first camera; constructing a pixel mapping of the projector space of the projector visible to the third camera and the second camera, and obtaining a pixel mapping between the second camera and the third camera; based on the pixel mapping between the second camera and the first camera, and the pixel mapping between the second camera and the third camera, obtaining a pixel mapping between the third camera and the first camera; based on the pixel mapping between the second camera and the first camera, establishing pixel mappings between the remaining cameras and the first camera one by one, based on the constructed pixel mappings, and obtaining mutual mappings between each camera and the first camera;

[0011] The pixel correspondence between the receiving surface model and the projector space is obtained based on the pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space, including: recording the projectors as current projectors one by one, determining a camera that can shoot the projection light path of the current projector, recording it as the current camera, projecting the pixels of the receiving surface model to the first camera space, and then using the mutual mapping to project to the current camera space; using the pixel mapping between the current projector and the current camera to project to the current projector space.

[0012] Furthermore, more than one camera is included, and at least one camera is set with multiple camera positions, and in different camera positions, the camera corresponds to different areas of the receiving surface;

[0013] The constructing of the pixel mapping of the camera space and the projector space, and the pixel mapping of the camera space and the receiving surface model space, includes: constructing a pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position, and constructing a pixel mapping between the camera space and the receiving surface model space at the first camera position; moving the camera to the second camera position, and constructing a pixel mapping of the projector space of the projector visible to both the second camera position and the first camera position, and obtaining a pixel mapping between the camera space at the second camera position and the first camera position. Projection; moving the camera to a third camera position, constructing a pixel mapping of the projector space of the projector visible to both the third camera position and the second camera position, and obtaining a pixel mapping between the second camera position and the third camera position; obtaining a pixel mapping between the third camera position and the first camera position based on the pixel mapping between the second camera position and the first camera position, and the pixel mapping between the second camera position and the third camera position; establishing pixel mappings between the camera spaces at the remaining camera positions and at the first camera position one by one based on the constructed pixel mapping, and obtaining a mutual mapping between the camera spaces at each camera position and at the first camera position;

[0014] The pixel correspondence between the connecting surface model and the projector space is obtained according to the pixel mapping between the camera space and the projector space and between the camera space and the connecting surface model space, including: recording the projectors one by one as the current projector, determining the camera position that can completely shoot the projection light path of the current projector, recording it as the current position, projecting the pixels of the connecting surface model to the camera space at the first position, and then using the mutual mapping to project to the camera space of the current position; using the pixel mapping between the current projector and the current camera to project to the current projector space.

[0015] Furthermore, the pixel mapping between the camera space and the projector space is constructed based on the structured light sequence graph generated by the projector, including:

[0016] The projector projects a structured light sequence diagram on a receiving surface; the camera corresponding to the projector synchronously captures each structured light sequence diagram, and then applies a threshold to convert each captured structured light sequence diagram into a binary sequence diagram, and decodes the encoding value of each pixel in the binary sequence diagram; finds the pixel points that are the same as the coding in the projector space, marks them as paired pixels, and the paired pixels are pixel mappings of the projector space and the camera space.

[0017] Furthermore, it comprises a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface;

[0018] The constructing of the pixel mapping of the camera space and the projector space and the pixel mapping of the camera space and the receiving surface model space includes: constructing a pixel mapping between the first camera space and the projector space of the projector visible to the first camera, which is recorded as pixel mapping one, and constructing a pixel mapping between the first camera and the receiving surface model, which is recorded as pixel mapping two; the first camera is one of the multiple cameras; constructing a pixel mapping between another camera and any of the projectors visible to the first camera, which is recorded as pixel mapping three; the another camera is another camera among the multiple cameras except the first camera;

[0019] Based on the pixel mapping one and the pixel mapping two, a pixel mapping between the 2D coordinates of each pixel point of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained, which is recorded as pixel mapping four;

[0020] Based on the pixel mapping three and the pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, which is recorded as pixel mapping five;

[0021] Based on the point pair relationship of the pixel mapping five, construct an overall pixel mapping of the camera space of the other camera and the receiving surface model space, which is recorded as pixel mapping six;

[0022] According to the construction method of the mapping three to the mapping six, a pixel mapping between the camera space of each of the remaining cameras and the receiving surface model space is constructed;

[0023] The obtaining the pixel correspondence between the connecting surface model and the projector space according to the pixel mapping between the camera space, the projector space and the connecting surface model space comprises:

[0024] The projectors are recorded as current projectors one by one, a camera that can completely capture the projection light path of the current projector is determined and recorded as the current camera, and each pixel of the receiving surface model is projected into the current camera space using pixel mapping between the current camera and the receiving surface model space;

[0025] Projects into the current projector space, using a pixel map between the current camera and this projector.

[0026] Furthermore, the receiving surface model includes any one of a plane, an arc surface, and an irregular surface.

[0027] To solve the above technical problems, this application provides another technical solution:

[0028] An automatic correction system for fused projection, wherein the fused projection is jointly projected by two or more projectors, and the automatic correction system comprises:

[0029] A modeling module, used for modeling the receiving surface of the projection to obtain a receiving surface model;

[0030] A pixel mapping module, used to construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space;

[0031] A correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space, the projector space and the receiving surface model space; and map the content to be projected to the projector space according to the pixel correspondence.

[0032] Furthermore, the automatic correction system includes a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface;

[0033] The pixel mapping module is used to construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the interface model space, including: constructing a pixel mapping between the first camera space and the projector space of the projector visible to the first camera, and constructing a pixel mapping between the first camera and the interface model space; the first camera is one of the multiple cameras; constructing a pixel mapping of the projector space of the projector visible to the second camera and the first camera, and obtaining a pixel mapping between the second camera and the first camera; constructing a pixel mapping of the projector space of the projector visible to the third camera and the second camera, and obtaining a pixel mapping between the second camera and the third camera; based on the pixel mapping between the second camera and the first camera, and the pixel mapping between the second camera and the third camera, obtaining a pixel mapping between the third camera and the first camera; based on the pixel mapping between the second camera and the first camera, establishing pixel mappings between the remaining cameras and the first camera one by one, and obtaining mutual mappings between each camera and the first camera;

[0034] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space according to the pixel mapping between the camera space and the projector space and the camera space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including: recording the projectors one by one as the current projector, determining the camera that can shoot the projection light path of the current projector, recording it as the current camera, projecting the pixels of the receiving surface model to the first camera space, and then using the mutual mapping to project it to the current camera space; using the pixel mapping between the current projector and the current camera to project it to the current projector space.

[0035] Furthermore, the automatic correction system includes more than one camera, at least one of which is provided with multiple camera positions, and at different camera positions, the camera corresponds to different areas of the receiving surface;

[0036] The pixel mapping module is used to construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space, including: constructing a pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position, and a pixel mapping between the camera space and the receiving surface model space at the first camera position; moving the camera to the second camera position, and constructing a pixel mapping between the camera and the projector space of the visible projector at the second camera position, and obtaining a pixel mapping between the camera space at the second camera position and the first camera position. Pixel mapping; moving the camera to a third camera position, constructing a pixel mapping of the projector space of the projector visible to both the third camera position and the second camera position, and obtaining a pixel mapping between the second camera position and the third camera position; obtaining a pixel mapping between the third camera position and the first camera position based on the pixel mapping between the second camera position and the first camera position, and the pixel mapping between the second camera position and the third camera position; establishing pixel mappings between the camera spaces at the remaining camera positions and at the first camera position one by one based on the constructed pixel mapping, and obtaining a mutual mapping between the camera spaces at each camera position and at the first camera position;

[0037] The correction module is used to obtain the pixel correspondence between the receiving surface model and the projector space according to the pixel mapping between the camera space and the projector space and the camera space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including: recording the projectors as the current projectors one by one, determining the camera position that can completely capture the projection light path of the current projector, recording it as the current position, projecting the pixels of the receiving surface model to the camera space at the first position, and then using the mutual mapping to project it to the camera space of the current position; using the pixel mapping between the current projector and the current camera to project it to the current projector space.

[0038] Furthermore, the automatic correction system includes a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface; the pixel mapping module is used to construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space, including:

[0039] Constructing a pixel mapping between a first camera space and a projector space of the projector visible to the first camera, which is recorded as pixel mapping one, and constructing a pixel mapping between the first camera and the receiving surface model, which is recorded as pixel mapping two; the first camera is one of the multiple cameras;

[0040] Constructing a pixel mapping between another camera and any of the projectors visible to the first camera, which is recorded as pixel mapping three; the another camera is another camera among the multiple cameras except the first camera;

[0041] Based on the pixel mapping one and the pixel mapping two, a pixel mapping between the 2D coordinates of each pixel point of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained, which is recorded as pixel mapping four;

[0042] Based on the pixel mapping three and the pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, which is recorded as pixel mapping five;

[0043] Based on the point pair relationship of the pixel mapping five, construct an overall pixel mapping of the camera space of the other camera and the receiving surface model space, which is recorded as pixel mapping six;

[0044] According to the construction method of mapping three to mapping six, a pixel mapping between the camera space of each of the remaining cameras and the receiving surface model space is constructed.

[0045] The correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space, the projector space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including:

[0046] The projectors are recorded as current projectors one by one, a camera that can completely capture the projection light path of the current projector is determined and recorded as the current camera, and each pixel of the receiving surface model is projected into the current camera space using pixel mapping between the current camera and the receiving surface model space;

[0047] Projects into the current projector space, using a pixel map between the current camera and this projector.

[0048] Different from the prior art, the automatic correction method and system of the fusion projection of the above technical solution constructs the pixel mapping of the camera space and the projector space, as well as the pixel mapping of the camera space and the receiving surface model space in the fusion projection. Therefore, according to the pixel mapping of the camera space, the projector space and the receiving surface model space, the pixel correspondence between the receiving surface model and the projector space can be obtained; therefore, the content to be projected can be mapped to the projector space according to the pixel correspondence, thereby realizing the automatic correction of the projector to the projection receiving surface during multi-machine fusion projection, greatly improving the efficiency and accuracy of multi-machine fusion projection correction. In addition, in the above scheme, the technology of multiple cameras or one machine with multiple positions in time is used to expand the field of view of the automatic correction, and a larger receiving surface can be corrected. And when multiple cameras are corrected, the structured light projected by the projector is used to establish the connection feature points between the cameras, so that the correction is more accurate and the calculation amount is smaller.

[0049] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present invention and other related contents, and shall not be considered as limiting the present application.

[0051] In the drawings of the specification:

[0052] Figure 1 It is a flowchart of the automatic correction method of the fused projection described in the specific implementation method;

[0053] Figure 2 It is a schematic diagram of automatic correction of fused projection according to a specific implementation method;

[0054] Figure 3 It is a flowchart of a method for automatic correction of fusion projection of multiple cameras in a specific implementation mode;

[0055] Figure 4 It is a schematic diagram of automatic correction of fusion projection of multiple cameras according to a specific implementation method;

[0056] Figure 5 A schematic diagram of a two-camera and two-projector calibration using a building as a receiving surface for a specific implementation method;

[0057] Figure 6 It is a flowchart of another embodiment of a method for automatic correction of multi-camera fusion projection;

[0058] Figure 7 It is a module block diagram of the automatic correction system of fused projection described in the specific implementation method;

[0059] The reference numerals in the above drawings are described as follows:

[0060] 700. Automatic correction system for fused projection; 701. Modeling module; 702. Pixel mapping module; 703. Correction module. DETAILED DESCRIPTION

[0061] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0062] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0063] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0064] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0065] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0066] Without further limitations, in this application, the words "include", "comprises", "has" or other similar open-ended expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0067] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0068] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0069] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0070] See also Figure 1, this embodiment provides an automatic correction method for fused projection. In the automatic correction method for fused projection, the fused projection refers to the use of more than two projectors to jointly perform projection, wherein different projectors are responsible for the projection of different areas of the receiving surface, thereby achieving a larger area of ​​projection. Take dual-projector fused projection as an example, wherein projector one can be used for the projection of the left area of ​​the receiving surface, and projector two is used for the projection of the right area of ​​the receiving surface, and the projection of the left area and the projection of the right area of ​​the receiving surface are spliced ​​to form a complete large-area projection. In this embodiment, the automatic correction method for fused projection is to automatically correct each projector in the upper fused projection, so that each projector can correct the content to be projected according to the relative position of the projector and the receiving surface and the shape of the receiving surface.

[0071] like Figure 1 As shown, in this embodiment, the automatic correction method of the fused projection includes the following steps:

[0072] S101, modeling the receiving surface of the projection to obtain a receiving surface model;

[0073] S102, constructing a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space;

[0074] S103, obtaining a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space, the projector space and the receiving surface model space; and mapping the content to be projected to the projector space according to the pixel correspondence.

[0075] In step S101, the projected receiving surface is modeled to obtain a receiving surface model including vertices and UV coordinates. The UV coordinates are a two-dimensional coordinate system, where U represents the horizontal direction and V represents the vertical direction. U and V together constitute a two-dimensional UV coordinate system.

[0076] In step S102, a pixel mapping of the camera space and the projector space can be constructed by structured light, and a pixel mapping of the camera space and the receiving surface model space can be constructed by feature point matching. In this step, the structured light is generated by the projector and projected onto the receiving surface. The projector can project a structured light sequence diagram, and the camera corresponding to the projector synchronously captures each structured light sequence diagram, and then applies a threshold to convert each structured light sequence diagram into a binary sequence diagram, decodes the encoding value of each pixel in the binary sequence diagram, finds the pixel points that are the same as the projector space and the encoding, and marks them as paired pixels, that is, the pixel mapping of the projector space and the camera space. The projector projects a structured light pattern on the receiving surface; the camera corresponding to the projector synchronously captures the structured light pattern, and then aligns and decodes it; according to the pixel encoding, the acceptance pixels of the camera space and the projector space are found, and the paired pixels are the pixel mapping of the projector space and the camera space.

[0077] In step S103, each pixel coordinate in the receiving surface model is projected to the camera space through the pixel mapping between the camera space and the receiving surface model space obtained in step S102; and then projected to the projector space through the pixel mapping between the camera space and the projector space obtained in step S102. Finally, the content to be projected (such as a video) is mapped to the projector space according to the correspondence between the UV coordinates and the pixels of the receiving surface model. It should be noted that in this embodiment, the pixels of the content to be projected are mapped to the receiving surface model according to the correspondence between the UV coordinates and the pixels of the receiving surface model; then, according to the pixel mapping between the camera space and the receiving surface model space, they are mapped to the camera space, and finally, according to the pixel mapping between the camera space and the projector space, they are mapped to the projector space.

[0078] The automatic correction method of the fused projection can be applied to single-camera application scenarios as well as multi-camera application scenarios. When a single camera can capture the entire projection receiving surface, only one camera is needed to implement the automatic correction method of the fused projection. Figure 2 As shown, it is a schematic diagram of automatic correction of fusion projection of dual projectors using a single camera.

[0079] When the field of view of a single camera cannot cover the entire projection receiving surface, multiple cameras are used to perform the above-mentioned automatic correction method of fusion projection. Each camera is used to capture images of different areas of the receiving surface. Figure 4 , which is a schematic diagram of automatic correction of fusion projection of dual projectors using two cameras; Figure 4 In the figure, the camera on the lower left (camera 1) is used to collect the receiving surface ( Figure 4The other camera at the lower right (camera 2) is used to capture the image on the right side of the receiving surface, the projector at the upper left (projector 1) is used to project the image on the left side of the receiving surface, and the other projector at the upper right (projector 2) is used to project the image on the right side of the receiving surface.

[0080] exist Figure 2 and Figure 4 In the figure, the green arrow line shows the pixel mapping process of the projector in the upper left corner; the red arrow line shows the pixel mapping process of the projector 2 in the upper right corner. The above arrow lines only indicate the process, not the order of the operation steps. For example, Figure 4 For the case of a planar dual-camera dual-projector, the two red lines on the left are operated the same as the green lines and can be completed in the same step. The difference between the automatic calibration of the fused projection using a single camera and the automatic calibration of the fused projection using multiple cameras is mainly in step S103. When using a single camera for calibration, it is only necessary to establish a pixel mapping between the camera and the receiving surface model; but when using multiple cameras for calibration, it is necessary to establish a pixel mapping between each camera and each corresponding area of ​​the receiving surface model.

[0081] In this embodiment, according to the pixel mapping of the camera space, the projector space and the receiving surface model space, the pixel correspondence between the receiving surface model and the projector space can be obtained; therefore, the content to be projected can be mapped to the projector space according to the pixel correspondence, thereby realizing automatic correction of the projector to the projection receiving surface during multi-machine fusion projection, greatly improving the efficiency and accuracy of multi-machine fusion projection correction.

[0082] like Figure 3 As shown, in this embodiment, a method for automatically correcting fused projection using multiple cameras is provided. The method for automatically correcting fused projection includes multiple cameras, each camera corresponding to a different area of ​​the receiving surface; the method for automatically correcting fused projection includes the following steps:

[0083] S301, modeling the receiving surface of the projection to obtain a receiving surface model;

[0084] S302, constructing a pixel mapping between a first camera space and a projector space of the projector visible to the first camera, and constructing a pixel mapping between the first camera and a receiving surface model space; the first camera is one of the plurality of cameras;

[0085] S303, constructing a pixel mapping of the projector space of the projector visible to the second camera and the first camera, and obtaining a pixel mapping between the second camera and the first camera; constructing a pixel mapping of the projector space visible to the third camera and the second camera, and obtaining a pixel mapping between the second camera and the third camera; obtaining a pixel mapping between the third camera and the first camera based on the pixel mapping between the second camera and the first camera and the pixel mapping between the second camera and the third camera; establishing pixel mappings between the remaining cameras and the first camera one by one based on the constructed pixel mappings, and obtaining mutual mappings between each camera and the first camera;

[0086] S304, recording the projectors one by one as current projectors, determining a camera that can capture the projection light path of the current projector, recording the camera as the current camera, projecting pixels of the receiving surface model to the first camera space, and then projecting them to the current camera space using the mutual mapping;

[0087] S305: Use pixel mapping between the current projector and the current camera to project to the current projector space.

[0088] In this embodiment, the pixel mapping relationship between the receiving surface model and the projector is realized by constructing a camera mutual mapping between multiple cameras. In step S302, the pixel mapping between the first camera space and the projector space can be constructed by structured light, and it is marked as pixel mapping a; the pixel mapping between the first camera and the receiving surface model space is constructed by feature point matching, and it is marked as pixel mapping b. In step S303, for the second camera, the pixel mapping between the second camera space and the space of a visible projector in the first camera is constructed by structured light, which is recorded as pixel mapping c; the pixel mapping between the second camera space and the first camera space is constructed by pixel mapping a and pixel mapping c. Similarly, if there is a third camera, the pixel mapping between the third camera space and the second camera can be constructed, and so on, to construct the mutual mapping between all cameras and the first camera. In step S303, the mapping between the second camera and the first camera is constructed by a certain projector. The mutual mapping between other cameras and the first camera is further constructed based on the constructed pixel mapping. For example:

[0089] Constructing a mapping 1, where the mapping 1 is a mapping between a first camera and a projector;

[0090] Constructing a mapping 2, where the mapping 2 is a mapping between the second camera and the projector;

[0091] The pixel mapping between the first camera and the second camera can be obtained through mapping 1 and mapping 2.

[0092] The pixel mapping between the third camera and the first camera is achieved by:

[0093] Construct mapping 3, which is the mapping between the second camera and a projector.

[0094] Constructing a mapping 4, where the mapping 4 is a mapping between a third camera and a projector;

[0095] The pixel mapping between the third camera and the second camera can be obtained through mapping 3 and mapping 4, and then the mutual mapping between the third camera and the first camera can be obtained through the previously constructed pixel mapping between the first camera and the second camera.

[0096] In other words, except for the second camera which is directly mapped with the first camera, the mappings between other cameras and the first camera are constructed step by step based on the constructed pixel mapping.

[0097] In step S304 and step S305, when any projector is recorded as the current projector, a camera that can completely capture the projection light path of the current projector is found and recorded as the current camera. First, pixel mapping b is applied to project the surface pixels of the receiving surface model to the first camera, and then the mutual mapping between the first camera and the current camera is applied to project to the current camera; then the pixel mapping between the projector and the current camera is applied to project to the projector.

[0098] In another embodiment, a camera is moved to different positions multiple times in time division to replace the above-mentioned automatic correction method using multiple cameras for fusion projection. In this automatic correction method, one or more cameras can be used, and at least one of the cameras is set to multiple positions. At different positions, the camera corresponds to different areas of the receiving surface. For example, camera 1 has three positions. At the first position, the camera shoots the area on the left side of the receiving surface. At the second position, the camera shoots the area in the middle of the receiving surface. At the third position, the camera shoots the area on the right side of the receiving surface. Therefore, the three positions of a camera can cover the receiving surface that could not be fully photographed before.

[0099] The automatic correction method of the fusion projection using multiple camera positions moving to different camera positions multiple times in time-sharing includes:

[0100] Constructing a pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position, and constructing a pixel mapping between the camera space at the first camera position and the receiving surface model space;

[0101] Move the camera to a second camera position, and construct a pixel mapping of the projector space of the projector visible to both the second camera position and the first camera position, so as to obtain a pixel mapping between the camera space at the second camera position and the first camera position; move the camera to a third camera position, and construct a pixel mapping of the projector space visible to both the third camera position and the second camera position, so as to obtain a pixel mapping between the second camera position and the third camera position; based on the pixel mapping between the second camera position and the first camera position, and the pixel mapping between the second camera position and the third camera position, obtain a pixel mapping between the third camera position and the first camera position; based on the constructed pixel mapping, establish pixel mappings between the camera space at the remaining camera positions and the first camera position one by one, so as to obtain a mutual mapping between the camera space at each camera position and the first camera position;

[0102] The projectors are recorded as current projectors one by one, a camera position that can capture the projection light path of the current projector is determined, recorded as the current camera position, pixels of the receiving surface model are projected to the camera space of the first camera position, and then projected to the camera space of the current camera position using the mutual mapping;

[0103] Projecting to the current projector space using a pixel mapping between the current projector and the current camera.

[0104] In this embodiment, fewer cameras can be used to shoot the entire receiving surface and complete the automatic correction of the fusion projection. In this embodiment, one camera can be used to replace all cameras, that is, only one camera is needed. For example, in the case where three cameras are originally used for automatic correction, only one camera is used, and the camera has three positions, and the shooting range of each position corresponds to the original camera. In other embodiments, one or two cameras can also be used to replace some of the cameras in the above embodiments. For example, in the case where three cameras are originally used for automatic correction, one camera (with two different positions) can be used to replace two of the cameras, and the third camera can be retained. For another example, in the case where four cameras are originally used for automatic correction, two cameras can also be used as a replacement, each camera having two positions, and each camera replaces the original two cameras.

[0105] like Figure 6 As shown, in another embodiment, another automatic correction method for fusion projection using multiple cameras is provided. In this embodiment, a plurality of the cameras are included, each camera corresponds to a different area of ​​the receiving surface, and multi-camera correction is achieved by constructing a pixel mapping between each camera and the receiving surface model.

[0106] like Figure 6 As shown, the automatic correction method for fusion projection using multiple cameras in this embodiment includes:

[0107] S601, constructing a pixel mapping of a first camera space and a visible projector by structured light, recorded as pixel mapping 1;

[0108] S602, constructing a pixel mapping of the first camera space and the receiving surface space by matching feature points, recorded as pixel mapping 2;

[0109] S603. For the second camera, a visible projector in the first camera is visible, and the projector is recorded as projector one. A pixel mapping between the second camera space and the projector one space is constructed by structured light, and is recorded as pixel mapping three. The second camera can see a visible projector in the first camera, which means that the second camera can capture a part of the light path of the projector captured by the first camera. For example, the first camera can capture projector 1, and the second camera can capture a part of the light path of projector 1.

[0110] S604, based on pixel mapping one and pixel mapping two, a pixel mapping of 2D coordinates of each pixel point of projector one and 3D coordinates of each point projected onto the receiving surface model can be obtained, which is recorded as pixel mapping four;

[0111] S605, based on the pixel mapping three and the pixel mapping four, a partial pixel mapping of the 3D points of the second camera and the receiving surface model can be obtained, which is recorded as pixel mapping five;

[0112] S606, constructing an overall pixel mapping of the second camera space and the receiving surface model space based on the point pair relationship of the pixel mapping five, recorded as pixel mapping six;

[0113] S607, according to steps S603-S606, calculating pixel mappings of the remaining cameras and the receiving surface model space;

[0114] S608. For any projector, find a camera that can completely capture the projection light path of the projector, record it as the current camera, apply the pixel mapping between the current camera and the space of the receiving surface model, project the surface pixels of the receiving surface model to the current camera space; then apply the pixel mapping between the current camera space and the projector space, project it to the projector. Through step S608, the projectors can be recorded as the current projector one by one, determine the camera that can completely capture the projection light path of the current projector, record it as the current camera, use the pixel mapping between the current camera and the space of the receiving surface model, project each pixel of the receiving surface model to the current camera space; and use the pixel mapping between the current camera and the projector to project it to the current projector space.

[0115] like Figure 2 , Figure 4 and Figure 5 As shown, in the above embodiment, the projection receiving surface can be any one of a plane, an arc surface, and an irregular surface. Figure 2The bearing surface shown in is a cylindrical surface (i.e. a type of arc surface). Figure 4 The receiving surface shown is a rectangular receiving surface (i.e. a type of plane), Figure 5 The receiving surface shown is the building facade (i.e., a type of irregular surface).

[0116] like Figure 2 The figure shows a schematic diagram of automatic correction of fused projection of a cylindrical single camera and dual projectors. The method for automatic correction of fused projection on a cylindrical surface includes the following steps:

[0117] 11. Place the camera so that it can capture the entire cylinder (i.e. the projection surface).

[0118] 12. Modeling the cylinder: Measure the shape parameters of the cylinder and model it. When modeling, take the upper left corner of the cylinder as the UV coordinate (0,0) and the lower right corner as the UV coordinate (1,1) to obtain the UV coordinate.

[0119] 13. Calculate the pixel mapping of the camera space and the projector space: In this embodiment, a binary structured light algorithm is selected to construct the pixel mapping of the camera and the projector. The first projector projects a structured light sequence image, and the camera synchronously captures each structured light sequence image. A threshold is applied to convert each structured light sequence image into a binary image. The encoding of the projected pixel that matches the binary sequence of each pixel is found and marked as a paired pixel, that is, the pixel mapping of the first projector space and the camera space. Similarly, the pixel mapping of the second projector space and the camera space can be calculated.

[0120] 14. Calculate the pixel mapping between the camera space and the receiving surface (cylinder) space: The camera shoots the cylinder, and the edge detection algorithm is used to extract the edge points of the cylinder. These edge points are fitted with a Bezier surface to represent the shape of the cylinder projected onto the camera imaging surface.

[0121] 15. Project cylindrical pixels to projector space: For any cylindrical pixel, its UV coordinates can be found, and the UV coordinates can be substituted into the Bezier surface formula to obtain its 2D coordinates projected to the camera space; combined with the pixel mapping of the camera space and the projector space in step 13, the corresponding projector can be queried, that is, the 2D coordinates of the projector space.

[0122] 16. Video is mapped to projector space by UV: Get a video, which is a planar video made by cylindrical UV expansion. For any pixel point on the video, substitute its UV into the model in step 12 to get the model pixel coordinates, and substitute it into step 15 to project it to each projector space. Project all video pixels to get the corrected image.

[0123] like Figure 4The figure is a schematic diagram of automatic correction of fusion projection of dual cameras and dual projectors with a plane as the receiving surface. The automatic correction method of fusion projection of dual cameras and dual projectors includes the following steps:

[0124] 21. Place the cameras so that Camera 1 captures the left side of the receiving surface and Camera 2 captures the right side of the receiving surface; place the projectors so that Projection 1 projects on the left side of the receiving surface and Projection 2 projects on the right side of the receiving surface. Adjust Camera 2 so that Camera 2 can capture part of the light path of Projector 1.

[0125] 22. Modeling: Measure the width and height of the receiving surface, build a model of the receiving surface, and expand it according to the upper left corner (0,0) and the lower right corner (1,1) of the receiving surface to obtain the UV coordinates.

[0126] 23. Calculate the pixel mapping of the camera space and the projector space: Projector 1 projects a sequence of structured light images, which are captured and recognized by cameras 1 and 2 simultaneously, and the pixel mapping A between the projector 1 space and the camera 1 space, and the pixel mapping B between the projector 1 space and the camera 2 space, are obtained. Substitute the point pairs of pixel mapping A into the least squares method to solve the homography matrix A between projector 1 and camera 1; similarly, the homography matrix B between projector 1 and camera 2 is obtained. Projector 2 projects a sequence of structured light images, which are captured and recognized by camera 2, and the homography matrix is ​​solved to obtain the homography matrix C between projector 2 and camera 2.

[0127] 24. Calculate the pixel mapping between the camera space and the receiving surface (plane) space: mark 4 points on the receiving surface, and obtain the 4 points corresponding to the marked points in camera one, and use these 4 point pairs to solve the homography matrix D, that is, the pixel mapping between the camera one space and the receiving surface space.

[0128] 25. The receiving surface pixel points are projected to projector one (pixel mapping C): Get the pixel points of the receiving surface, apply the homography matrix D, map them to the camera one space, and then apply the homography matrix A to map them to the projection one space.

[0129] 26. Plane pixel points are projected to projector 2 (pixel mapping D): Get the pixel points of the receiving surface, apply the homography matrix D, map it to camera 1 space; then apply the homography matrix A, map it to projection 1 space; then apply the homography matrix B to map it to camera 2 space; then apply the homography matrix C to map it to projection 2 space.

[0130] 27. Video mapping to projector space: Get a video that is produced by plane UV expansion. For any pixel, query its corresponding receiving surface pixel point with UV coordinates, apply pixel mapping C or pixel mapping D to project it to the corresponding projector space, and you can get the corrected image.

[0131] like Figure 5As shown, it is a schematic diagram of automatic correction of fusion projection of dual cameras and dual projectors whose receiving surface is a building. In the figure, "a" represents pixel mapping a; "b" represents pixel mapping b; "c" represents pixel mapping c; "d" represents pixel mapping d; "e" represents pixel mapping e; "f" represents pixel mapping f; "g" represents pixel mapping g, and four circles represent four vertices that are easily identifiable on the receiving surface. The automatic correction method of fusion projection of dual cameras and dual projectors whose receiving surface is a building includes the following steps:

[0132] 31. Use Zhang Zhengyou's calibration method to calibrate the intrinsic parameters and distortion coefficients of camera 1 and camera 2.

[0133] 32. Place the cameras so that Camera 1 captures the left side of the building and Camera 2 captures the right side of the building; place the projectors so that Projection 1 projects onto the left side of the building and Projection 2 projects onto the right side of the building. Adjust Camera 2 so that it can capture part of the light path of Projector 1.

[0134] 33. Connecting surface modeling: Use laser scanning to construct the connecting surface model and correctly unfold it in 3D software to obtain UV coordinates.

[0135] 34. Calculate the pixel mapping of the camera and projector: Projection 1 projects a structured light sequence, which is captured and recognized by camera 1 and camera 2 simultaneously, and obtains the pixel mapping a between the projection 1 space and the camera 1 space, and the pixel mapping b between the projection 1 space and the camera 2 space. Projection 2 projects a structured light sequence, which is captured and recognized by camera 2, and obtains the pixel mapping c between the projection 2 space and the camera 2 space.

[0136] 35. Calculate the pixel mapping d between camera 1 and the interface (building): On the left side of the building, find four easily identifiable vertices (such as Figure 5 ), obtain its internal coordinates in camera one, and get 4 point pairs. The internal parameters of camera one are known, so the PnP algorithm can be used to solve the posture of camera one relative to the building (spatial coordinates, yaw angle (Yaw), pitch angle (Pitch), roll angle (Roll)). Given the posture, internal parameters, and distortion coefficients, the perspective matrix can be used to calculate the imaging point coordinates of each vertex of the model in camera one, and the pixel mapping of the camera one space and the receiving surface model can be obtained.

[0137] 36. Calculate the pixel mapping e between camera 2 and the receiving surface: Combine pixel mapping a and pixel mapping d to get the coordinates of each pixel of projection 1 on the receiving surface; Combine pixel mapping b to get the pixel mapping relationship between some pixels of camera 2 and the vertices of the receiving surface, that is, the point pair relationship between several points in the space of camera 2 and several points in the model space of the receiving surface. Combine the internal parameters of camera 2 and apply the PnP algorithm to solve the position and posture of camera 2 relative to the building.

[0138] 37. Project building vertices to projector one (pixel mapping f): Get the building vertex, apply pixel mapping d to get its coordinates in camera one; then apply pixel mapping a to get its coordinates in projection one.

[0139] 38. Project the building vertex to projector 2 (pixel map g): Get the building vertex, apply pixel map e to get its coordinates in camera 2; then apply pixel map c to get its coordinates in projector 2.

[0140] 39. Map the content to the projector space. There are two methods. Method 1: Look up the coordinates of the model vertices according to UV for each pixel of the video, and then apply pixel mapping f or pixel mapping g to project it to the projector.

[0141] Method 2 includes:

[0142] Step 391, calculate the pose of the projector: the projector can be expressed as a pinhole camera model in the mathematical model, so its internal parameters can also be expressed as focal length and optical center. From the pixel mapping f and g, the point pairs between the projector and the model can be extracted. In order to apply the PnP algorithm to solve the pose of the projector relative to the building, its internal parameters need to be solved. The traversal search method can be used to solve the internal parameters. The traversal search method to solve the internal parameters includes the following steps: regard the focal length and optical center internal parameters of the projector as dimensions of the multidimensional solution space, traverse and try each combination of the solution space, and under a specific combination, apply the PnP algorithm to solve the pose of the projector, and calculate the reprojection coordinates of the model vertices under this pose, and evaluate the reprojection error. After the complete traversal, select the dimension combination with the minimum error as the "equivalent internal parameter".

[0143] Step 392: After obtaining the intrinsic and extrinsic parameters of the projector, the perspective matrix is ​​applied to obtain the coordinates of each vertex of the model mapped to the projector. Compared with method 1, method 2 can overcome the problem of poor local correction caused by the recognition error of individual points of structured light.

[0144] In the above embodiments, the camera space, projector space, receiving surface space or receiving surface model space all refer to the mathematical space under its coordinate system; 2) Sometimes the word "space" is omitted in the expression, such as "projected to the first camera" actually means "projected to the first camera space".

[0145] like Figure 7 As shown, in another embodiment, a fusion projection automatic correction system 700 is provided. The fusion projection is projected by two or more projectors. The fusion projection automatic correction system 700 includes: a modeling module 701, a pixel mapping module 702 and a correction module 703.

[0146] The modeling module 701 is used to model the receiving surface of the projection to obtain a receiving surface model;

[0147] The pixel mapping module 702 is used to construct pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space;

[0148] The correction module 703 is used to obtain the pixel correspondence between the receiving surface model and the projector space according to the pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space; and map the content to be projected to the projector space according to the pixel correspondence.

[0149] In this embodiment, the receiving surface model includes any one of a plane, an arc surface, and an irregular surface.

[0150] The automatic correction system 700 of the fused projection can be applied to single-camera application scenarios as well as multi-camera application scenarios. When a single camera can capture the entire projected receiving surface, only one camera is needed to implement the above-mentioned automatic correction method of the fused projection. When the shooting field of view of a single camera cannot cover the entire projected receiving surface, multiple cameras are used to jointly perform the above-mentioned automatic correction method of the fused projection. Each camera is used to capture images of different areas of the receiving surface.

[0151] In one embodiment, the automatic correction system includes a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface;

[0152] The pixel mapping module is used to construct pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space, including:

[0153] Constructing a pixel mapping between a first camera space and a projector space of the projector visible to the first camera, and constructing a pixel mapping between the first camera and a receiving surface model space; the first camera is one of the plurality of cameras;

[0154] Constructing a pixel mapping of the projector space of the projector visible to the second camera and the first camera, and obtaining a pixel mapping between the second camera and the first camera; constructing a pixel mapping of the projector space visible to the third camera and the second camera, and obtaining a pixel mapping between the second camera and the third camera; obtaining a pixel mapping between the third camera and the first camera based on the pixel mapping between the second camera and the first camera and the pixel mapping between the second camera and the third camera; establishing pixel mappings between the remaining cameras and the first camera one by one based on the constructed pixel mappings, and obtaining mutual mappings between each camera and the first camera;

[0155] The correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including:

[0156] The projectors are recorded as current projectors one by one, a camera that can capture the projection light path of the current projector is determined and recorded as the current camera, pixels of the receiving surface model are projected into the first camera space, and then projected into the current camera space using the mutual mapping;

[0157] Projecting to the current projector space using a pixel mapping between the current projector and the current camera.

[0158] In this embodiment, automatic correction of multi-camera fusion projection is achieved by constructing mutual mapping between cameras.

[0159] In another embodiment, the automatic correction system includes more than one camera, and at least one camera is set at multiple positions, and at different positions, the camera corresponds to different areas of the receiving surface;

[0160] The pixel mapping module is used to construct pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space, including:

[0161] Constructing a pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position, and a pixel mapping between the camera space at the first camera position and the receiving surface model space;

[0162] Move the camera to a second camera position, and construct a pixel mapping of the projector space of the projector visible to both the second camera position and the first camera position, so as to obtain a pixel mapping between the camera space at the second camera position and the first camera position; move the camera to a third camera position, and construct a pixel mapping of the projector space visible to both the third camera position and the second camera position, so as to obtain a pixel mapping between the second camera position and the third camera position; based on the pixel mapping between the second camera position and the first camera position, and the pixel mapping between the second camera position and the third camera position, obtain a pixel mapping between the third camera position and the first camera position; based on the constructed pixel mapping, establish pixel mappings between the camera space at the remaining camera positions and the first camera position one by one, so as to obtain a mutual mapping between the camera space at each camera position and the first camera position;

[0163] The correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including:

[0164] The projectors are recorded as current projectors one by one, a camera position that can completely capture the projection light path of the current projector is determined, recorded as the current camera position, pixels of the receiving surface model are projected to the camera space of the first camera position, and then projected to the camera space of the current camera position using the mutual mapping;

[0165] Projecting to the current projector space using a pixel mapping between the current projector and the current camera.

[0166] In this embodiment, one camera is moved to different positions multiple times in time division to replace the above-mentioned multiple cameras for automatic correction of fusion projection. In this embodiment, fewer cameras can be used to shoot the entire receiving surface and complete the automatic correction of fusion projection.

[0167] In another embodiment, the automatic correction of the fusion projection of multiple cameras can be achieved by constructing a pixel mapping of each camera space and the receiving surface space. The automatic correction system includes a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface;

[0168] The pixel mapping module is used to construct pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space, including:

[0169] Constructing a pixel mapping between a first camera space and a projector space of the projector visible to the first camera, which is recorded as pixel mapping one, and constructing a pixel mapping between the first camera and the receiving surface model, which is recorded as pixel mapping two; the first camera is one of the multiple cameras;

[0170] Constructing a pixel mapping between another camera and any of the projectors visible to the first camera, which is recorded as pixel mapping three; the another camera is another camera among the multiple cameras except the first camera;

[0171] Based on the pixel mapping one and the pixel mapping two, a pixel mapping between the 2D coordinates of each pixel point of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained, which is recorded as pixel mapping four;

[0172] Based on the pixel mapping three and the pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, which is recorded as pixel mapping five;

[0173] Based on the point pair relationship of the pixel mapping five, construct an overall pixel mapping of the camera space of the other camera and the receiving surface model space, which is recorded as pixel mapping six;

[0174] Based on the pixel mapping three to the pixel mapping six, a pixel mapping between each camera space and the receiving surface model space is obtained;

[0175] The correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including:

[0176] The projectors are recorded as current projectors one by one, a camera that can completely capture the projection light path of the current projector is determined and recorded as the current camera, and each pixel of the receiving surface model is projected into the current camera space using pixel mapping between the current camera and the receiving surface model space;

[0177] Projects into the current projector space, using a pixel map between the current camera and this projector.

[0178] In the above embodiments, multiple cameras are used for automatic correction of fused projection, which expands the camera's shooting field of view during automatic correction, so that a larger receiving surface can be corrected. And when multiple cameras perform automatic correction of fused projection, structured light projected by the projector is used to establish the connection feature points between cameras, making the correction more accurate and less computationally intensive. In some embodiments, a single camera can be moved to different positions in time-sharing mode to replace multiple cameras, thereby reducing the number of cameras used and further improving applicability. And in the above embodiments, the type of receiving surface is not limited, and tasks such as simple planes and complex 3D Mapping are supported.

[0179] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. An automatic correction method for fused projection, characterized in that: The fusion projection uses more than two projectors to jointly project, and the automatic correction method includes the following steps: Modeling the receiving surface of the projection to obtain a receiving surface model; Constructing a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space; According to the pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space, the pixel correspondence between the receiving surface model and the projector space is obtained; and the content to be projected is mapped to the projector space according to the pixel correspondence.

2. The automatic correction method of fused projection according to claim 1, characterized in that: comprising a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface; The step of constructing pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space, comprises: Constructing a pixel mapping between a first camera space and a projector space of the projector visible to the first camera, and constructing a pixel mapping between the first camera and a receiving surface model space; the first camera is one of the plurality of cameras; Constructing a pixel mapping of the projector space of the projector visible to the second camera and the first camera, and obtaining a pixel mapping between the second camera and the first camera; constructing a pixel mapping of the projector space of the projector visible to the third camera and the second camera, and obtaining a pixel mapping between the second camera and the third camera; obtaining a pixel mapping between the third camera and the first camera based on the pixel mapping between the second camera and the first camera and the pixel mapping between the second camera and the third camera; Based on the constructed pixel mapping, pixel mappings between the remaining cameras and the first camera are established one by one to obtain mutual mappings between each camera and the first camera; The obtaining the pixel correspondence between the receiving surface model and the projector space according to the pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space comprises: The projectors are recorded as current projectors one by one, a camera that can capture the projection light path of the current projector is determined and recorded as the current camera, pixels of the receiving surface model are projected into the first camera space, and then projected into the current camera space using the mutual mapping; Projecting to the current projector space using a pixel mapping between the current projector and the current camera.

3. The automatic correction method of fused projection according to claim 1, characterized in that: The device comprises more than one camera, at least one of which is provided with multiple camera positions, and in different camera positions, the camera corresponds to different areas of the receiving surface; The step of constructing pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space, comprises: Constructing a pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position, and constructing a pixel mapping between the camera space at the first camera position and the receiving surface model space; Move the camera to a second camera position, and construct a pixel mapping of the projector space of the projector visible to both the second camera position and the first camera position, so as to obtain a pixel mapping between the camera space at the second camera position and the first camera position; move the camera to a third camera position, and construct a pixel mapping of the projector space visible to both the third camera position and the second camera position, so as to obtain a pixel mapping between the second camera position and the third camera position; based on the pixel mapping between the second camera position and the first camera position, and the pixel mapping between the second camera position and the third camera position, obtain a pixel mapping between the third camera position and the first camera position; based on the constructed pixel mapping, establish pixel mappings between the camera space at the remaining camera positions and the first camera position one by one, so as to obtain a mutual mapping between the camera space at each camera position and the first camera position; The obtaining the pixel correspondence between the receiving surface model and the projector space according to the pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space comprises: The projectors are recorded as current projectors one by one, a camera position that can completely capture the projection light path of the current projector is determined, recorded as the current camera position, pixels of the receiving surface model are projected to the camera space of the first camera position, and then projected to the camera space of the current camera position using the mutual mapping; Projecting to the current projector space using a pixel mapping between the current projector and the current camera.

4. The automatic correction method of fused projection according to claim 2 or 3, characterized in that: The pixel mapping between the camera space and the projector space is constructed based on the structured light sequence graph generated by the projector, including: The projector projects a structured light sequence diagram on the receiving surface; The camera corresponding to the projector synchronously captures each structured light sequence image, then applies a threshold to convert each captured structured light sequence image into a binary sequence image, and decodes the encoding value of each pixel in the binary sequence image; Pixels that are identical to the code in the projector space are found and marked as paired pixels, where the paired pixels are pixel mappings between the projector space and the camera space.

5. The automatic correction method of fused projection according to claim 1, characterized in that: comprising a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface; The step of constructing pixel mapping between the camera space and the projector space and pixel mapping between the camera space and the receiving surface model space includes: Constructing a pixel mapping between a first camera space and a projector space of the projector visible to the first camera, which is recorded as pixel mapping one, and constructing a pixel mapping between the first camera and the receiving surface model, which is recorded as pixel mapping two; the first camera is one of the multiple cameras; Constructing a pixel mapping between another camera and any of the projectors visible to the first camera, which is recorded as pixel mapping three; the another camera is another camera among the multiple cameras except the first camera; Based on the pixel mapping one and the pixel mapping two, a pixel mapping between the 2D coordinates of each pixel point of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained, which is recorded as pixel mapping four; Based on the pixel mapping three and the pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, which is recorded as pixel mapping five; Based on the point pair relationship of the pixel mapping five, construct an overall pixel mapping of the camera space of the other camera and the receiving surface model space, which is recorded as pixel mapping six; According to the construction method of the mapping three to the mapping six, a pixel mapping between the camera space of each of the remaining cameras and the receiving surface model space is constructed; The obtaining the pixel correspondence between the connecting surface model and the projector space according to the pixel mapping between the camera space, the projector space and the connecting surface model space comprises: The projectors are recorded as current projectors one by one, a camera that can completely capture the projection light path of the current projector is determined and recorded as the current camera, and each pixel of the receiving surface model is projected into the current camera space using pixel mapping between the current camera and the receiving surface model space; Projects into the current projector space, using a pixel map between the current camera and this projector.

6. The automatic correction method of fused projection according to claim 1, characterized in that: The receiving surface model includes any one of a plane, an arc surface, and an irregular surface.

7. An automatic correction system for fused projection, characterized in that: The fusion projection uses more than two projectors to jointly project, and the automatic correction system includes: A modeling module, used for modeling the receiving surface of the projection to obtain a receiving surface model; A pixel mapping module, used to construct a pixel mapping between the camera space and the projector space, and a pixel mapping between the camera space and the receiving surface model space; A correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space; and to map the content to be projected to the projector space according to the pixel correspondence.

8. The automatic correction system for fusion projection according to claim 7, characterized in that: The automatic correction system includes a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface; The pixel mapping module is used to construct pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space, including: Constructing a pixel mapping between a first camera space and a projector space of the projector visible to the first camera, and constructing a pixel mapping between the first camera and a receiving surface model space; the first camera is one of the plurality of cameras; Constructing a pixel mapping of the projector space of the projector visible to the second camera and the first camera, and obtaining a pixel mapping between the second camera and the first camera; constructing a pixel mapping of the projector space visible to the third camera and the second camera, and obtaining a pixel mapping between the second camera and the third camera; obtaining a pixel mapping between the third camera and the first camera based on the pixel mapping between the second camera and the first camera and the pixel mapping between the second camera and the third camera; establishing pixel mappings between the remaining cameras and the first camera one by one based on the constructed pixel mappings, and obtaining mutual mappings between each camera and the first camera; The correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including: The projectors are recorded as current projectors one by one, a camera that can capture the projection light path of the current projector is determined and recorded as the current camera, pixels of the receiving surface model are projected into the first camera space, and then projected into the current camera space using the mutual mapping; Projecting to the current projector space using a pixel mapping between the current projector and the current camera.

9. The automatic correction system for fusion projection according to claim 7, characterized in that: The automatic correction system comprises more than one camera, at least one of which is set at multiple positions, and at different positions, the camera corresponds to different areas of the receiving surface; The pixel mapping module is used to construct pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space, including: Constructing a pixel mapping between the camera space at the first camera position and the projector space of the projector visible to the camera at the current camera position, and a pixel mapping between the camera space at the first camera position and the receiving surface model space; Move the camera to a second camera position, and construct a pixel mapping of the projector space of the projector visible to both the second camera position and the first camera position, so as to obtain a pixel mapping between the camera space at the second camera position and the first camera position; move the camera to a third camera position, and construct a pixel mapping of the projector space visible to both the third camera position and the second camera position, so as to obtain a pixel mapping between the second camera position and the third camera position; based on the pixel mapping between the second camera position and the first camera position, and the pixel mapping between the second camera position and the third camera position, obtain a pixel mapping between the third camera position and the first camera position; based on the constructed pixel mapping, establish pixel mappings between the camera space at the remaining camera positions and the first camera position one by one, so as to obtain a mutual mapping between the camera space at each camera position and the first camera position; The correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space and the projector space and between the camera space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including: The projectors are recorded as current projectors one by one, a camera position that can completely capture the projection light path of the current projector is determined, recorded as the current camera position, pixels of the receiving surface model are projected to the camera space of the first camera position, and then projected to the camera space of the current camera position using the mutual mapping; Projecting to the current projector space using a pixel mapping between the current projector and the current camera.

10. The automatic correction system for fused projection according to claim 7, characterized in that: The automatic correction system includes a plurality of the cameras, each camera corresponding to a different area of ​​the receiving surface; The pixel mapping module is used to construct pixel mapping between the camera space and the projector space, and pixel mapping between the camera space and the receiving surface model space, including: Constructing a pixel mapping between a first camera space and a projector space of the projector visible to the first camera, which is recorded as pixel mapping one, and constructing a pixel mapping between the first camera and the receiving surface model, which is recorded as pixel mapping two; the first camera is one of the multiple cameras; Constructing a pixel mapping between another camera and any of the projectors visible to the first camera, which is recorded as pixel mapping three; the another camera is another camera among the multiple cameras except the first camera; Based on the pixel mapping one and the pixel mapping two, a pixel mapping between the 2D coordinates of each pixel point of the projector and the 3D coordinates of each point projected onto the receiving surface is obtained, which is recorded as pixel mapping four; Based on the pixel mapping three and the pixel mapping four, a partial pixel mapping of each point on the 3D coordinates of another camera and the receiving surface is obtained, which is recorded as pixel mapping five; Based on the point pair relationship of the pixel mapping five, construct an overall pixel mapping of the camera space of the other camera and the receiving surface model space, which is recorded as pixel mapping six; According to the construction method of the mapping three to the mapping six, a pixel mapping between the camera space of each of the remaining cameras and the receiving surface model space is constructed; The correction module is used to obtain a pixel correspondence between the receiving surface model and the projector space according to pixel mapping between the camera space, the projector space and the receiving surface model space, and map the content to be projected to the projector space according to the pixel correspondence, including: The projectors are recorded as current projectors one by one, a camera that can completely capture the projection light path of the current projector is determined and recorded as the current camera, and each pixel of the receiving surface model is projected into the current camera space using pixel mapping between the current camera and the receiving surface model space; Projects into the current projector space, using a pixel map between the current camera and this projector.

Images