Image acquisition methods in multi-projector systems

Abstract

A method of collecting structured light patterns in parallel in a multi-projector system is provided, including sequentially projecting images from each of a plurality of projectors onto a surface; capturing the sequentially projected images on the surface via at least one sensing device; creating one or more sensing device / projector masks for limiting the images projected by each of the plurality of projectors to portions that do not conflict with projected images projected by other projectors of the plurality of projectors; using the one or more projector masks to create a graph of projectors that project conflicting projections; and creating a plurality of aggregation groups from the graph, where projectors within a group do not conflict with each other, to simultaneously aggregate structured light patterns from the projectors of each of the aggregation groups.

Description

Technical Field

[0001] This invention relates to multi-projector systems, and more particularly to a method for collecting structured light patterns in parallel within a multi-projector system. Background Technology

[0002] Known structured light systems are used to project patterns (typically grids or horizontal stripes) onto a surface, causing the projected pattern to deform, and a vision system, including at least one sensing device such as a camera, to calculate depth and surface information of the pattern for projector calibration. In multi-projector systems, structured light is collected sequentially from the projectors, such that the total collection time is proportional to the number of projectors and the number of cameras. Existing structured light systems cannot quickly calibrate large-scale multi-projector systems. For example, in a system with 300 projectors, collecting structured light sequentially from all projectors at a 15-second collection sequence would take 75 minutes, an unacceptably long "downtime" for customers, developers, and installers attempting to troubleshoot the system. Summary of the Invention

[0003] One aspect of the present invention provides a method for unambiguous parallel collection of structured light patterns in a multi-projector system, without merging projection patterns from any projector, thereby solving the fundamental bottleneck problem of image acquisition in large-scale multi-projector systems.

[0004] Projectors in a multi-projector system may experience conflicts, such as due to overlap or indistinguishable patterns. While overlap can be a type of conflict, it is not always a conflict. For example, if each projector uses a different primary color (i.e., red / green / blue), three projectors can be arranged overlapping without conflict because the camera can clearly distinguish the patterns of each projector. Nevertheless, overlap is the most difficult type of conflict to control. Although different patterns and colors can be assigned to help identify and reduce conflict, overlap itself is predetermined by the projector arrangement.

[0005] In one aspect, a method for detecting conflicts is proposed, which involves capturing projected images using one or more sensing devices, or using other information (installation design information, previous calibration, etc.), and then reducing these conflicts by differentiating pattern / color assignments, separating / eliminating pattern / projector ambiguities within the image using various masks, colors, different patterns, etc., and then acquiring images based on the reduction of conflicts.

[0006] As described below, a process is proposed to identify overlapping charts for recognizing “first-round” conflicts. Based on this, colors and patterns can be assigned to projectors to reduce conflicts (e.g., two projectors may overlap but not conflict because they are assigned separate colors). The conflict-reduced charts can then be parsed into a small number of non-conflicting groups that can be aggregated simultaneously.

[0007] In one respect, the camera can use a mask to distinguish non-overlapping projectors. In other respects, the patterns used by each projector can be different, for example by the shape of the pattern (e.g., a pattern similar to a QR code, which is different for each projector, or one projector uses a square, another uses a triangle, etc.), and / or each projector can use different colors (i.e., one projects a red pattern, another projects a blue pattern, etc.), or the timing of the patterns can be different (including but not limited to subthreshold timing in the DMD bit plane) or other means, such that the projections from each projector can be distinguished and identified by the pattern (or a portion of the pattern).

[0008] In some embodiments, parallel image acquisition for a large-scale multi-projector system may include a) selectively masking camera images, b) decomposing the projector layout into projector graphs that conflict with each other due to, for example, overlapping or indistinguishable patterns, and therefore cannot be clustered simultaneously, c) determining the optimal clustering group of the layout (graph coloring), d) separating images from individual cameras into images from each projector based on the mask and pattern, and / or e) running the acquisition process in parallel across the projector group.

[0009] In a multi-projector system with multiple projectors, the above aspects can be achieved by a method for parallel acquisition of projected images projected onto a surface, comprising: creating one or more masks for limiting the acquisition of projected images projected by each of the multiple projectors by at least one sensing device to portions that do not overlap with projected images projected by other projectors in the multiple projectors; using the one or more masks to determine a graph of projectors whose images conflict; creating an aggregate group of multiple projectors whose images do not conflict from the graph; and simultaneously acquiring images from the projectors in each aggregate group.

[0010] These aspects and advantages, as well as other aspects and advantages, will subsequently become apparent, and are found in the details of the structure and operation described more fully below and claimed with reference to the accompanying drawings, which form part of this invention, wherein like reference numerals always denote like parts. Attached Figure Description

[0011] Figure 1 This is a block diagram of a multi-projector system according to one aspect of this specification.

[0012] Figure 2 Showing the use Figure 1 The system projects multiple images with overlapping portions for a single camera to capture.

[0013] Figures 3A-3C The use of a camera / projector mask for limitation was shown. Figure 1 In the system, the portion of the image projected by each projector that does not overlap with the images projected by other projectors, and Figure 3D The diagram shows the use of two cameras to collect structured light patterns from two clusters, using a camera / projector mask.

[0014] Figure 4 The image shows mask magnification based on one aspect to accommodate the movement of the camera and / or projector between different calibration runs.

[0015] Figure 5A and Figure 5B The example layout shows four overlapping projector images arranged into three clusters.

[0016] Figure 6A and Figure 6B The example layout shows six projectors with conflicting elements arranged into three clusters.

[0017] Figure 7 An example is shown where a complex layout of nine overlapping projectors is arranged into three clusters.

[0018] Figure 8 The image shows a two-dimensional array with overlapping tiled projectors arranged into four clusters.

[0019] Figure 9 The large dome display shows multiple overlapping projectors arranged in four clusters.

[0020] Figure 10 This is a flowchart illustrating, according to one aspect of the specification, the steps of a method for parallel collection of structured light patterns in a multi-projector system. Detailed Implementation

[0021] In this specification, the term "graph" is used to refer to any mathematical relationship between multiple projectors or between multiple projectors and one or more sensing devices. Examples of graphs include, but are not limited to, tables, data structures, Boolean true / false statements, matrices, lists of projectors and sensing devices, etc., to describe mathematical relationships prior to the simultaneous acquisition of projected images, as described below.

[0022] Figure 1This is a block diagram of a multi-projector system 100 according to one aspect of this specification. Multiple projectors 110A, 110B, 110C, etc., are arranged in a self-configurable, self-organizing network cluster and communicate with each other via a network 120 such as wireless Ethernet, RF, or infrared under the control of a processor 150. Those skilled in the art will understand that the processor 150 may include at least one processing element, typically a central processing unit (CPU) in the form of a microprocessor, and some type of computer memory, typically a semiconductor memory chip. The processing element performs arithmetic and logical operations, and the sorting and control unit can change the order of operations in response to stored information. Peripheral devices may be included, such as input devices (keyboard, mouse, joystick, etc.) and output devices (monitor screen, printer, etc.). Although in Figure 1 The example configuration shows three projectors, but it should be understood that large-scale multi-projector systems can include hundreds of such projectors.

[0023] When configured in a cluster, projectors 110A, 110B, and 110C display on display surface 140 or other physical objects what is perceived as a single, seamless composite image consisting of multiple projected images 200A, 200B, and 200C, such as... Figure 2 As shown. It should also be noted that the display surface 140 can be any smooth surface or a discontinuous and / or non-smooth surface, such as a statue, a castle, or multiple independent objects with space between them. The placement of the projectors 110A, 110B, and 110C can be arbitrary, i.e., the boundaries of the images do not need to be aligned.

[0024] like Figure 2 As shown, at least one camera 130A is provided having a field of view 210 for capturing projected images. According to one aspect, a parallel collection method is provided for concurrently collecting structured light patterns 200A, 200B, 200C by selectively masking camera images or other separations of projector patterns, and by decomposing the layout into a graph coloring problem, as described in detail below.

[0025] Determining which projectors overlap can be accomplished in several ways: In one method, each projector 110A, 110B, 110C, etc., is selectively turned on / off to determine where the projection is seen by one (or more) cameras, thereby creating a list of camera / projector masks 310A, 310B, 310C, etc., such as... Figure 3AAs shown. Optionally, additional available information, such as design information (the planned location where the projection wants to reach) or results of previous calibration, can be used to estimate and construct camera / projector masks. It should be noted that the masks do not need to be perfectly accurate—their estimates can be conservative, representing the possible locations where the projector might project. By using pixel masks for each projector, each camera, and / or different colors or patterns for each projector, pixels belonging to different projectors can be sensed in a single camera image (i.e., camera view 210) even if the exact projector pixel identification is uncertain. For example, in Figure 3B In the middle, mask 310B is selected to isolate structured light patterns 200A and 200C, while Figure 3C In this process, masks 310A and 310C are selected to isolate the structured light pattern 200B. Therefore, projector images can be acquired simultaneously using a single camera 130A, provided the projector images do not conflict. A single image from camera view 210 can then be used in multiple different projector decoding pipelines. The graphs with conflicting projectors are grouped into clusters using any suitable graph coloring algorithm, such as one based on node connectivity order, although greedy and heuristic algorithms can also be used, as described below.

[0026] In other embodiments, additional cameras may be included, such as Figure 1 Camera 130B is used. If the projector images do not conflict, both cameras 130A and 130B can be used to capture them simultaneously. For example, Figure 3D This demonstrates the use of two cameras to collect structured light patterns from two clusters, using the reference as described above. Figures 3A-3C The mask.

[0027] In some embodiments, masks 310A, 310B, and 310C can be enlarged to accommodate small variations in projection between different calibration runs, such as Figure 4 As shown.

[0028] According to the method described in this invention, non-overlapping images from multiple projectors can be simultaneously acquired by one or more cameras. Furthermore, the number of projectors that can simultaneously acquire images is limited by the number of projector image collisions. Therefore, as... Figure 5A and 5B As shown in the example, the four projector images 510A, 510B, 510C, and 510D collide to form three clusters (510A + 510C, 510B, and 510D), where Figure 5B The double arrows in the node diagram represent projector image collisions.

[0029] In any projector layout where connection nodes represent projector image conflicts, the minimum collection time is given by the minimum number of aggregation groups. Therefore, for Figure 6A and 6BFor example, the layout of six projectors 610A, 610B, 610C, 610D, 610E, and 610F can be arranged into three clusters 620A, 620B, and 610C, where non-conflicting projectors are grouped into group 1 (620A) = projectors 610A and 610E, group 2 (620B) = projectors 610C, 610D, and 610F, and group 3 (610C) = projector 610B. Figure 5B The same as in the node diagram chart, Figure 6A and 6B The double arrows in the image indicate projector image conflict.

[0030] In practice, complex arrangements of multiple projectors can be grouped based on projector conflicts to minimize the collection time of multi-projector systems. For example, in Figure 7 In this case, the complex layout of nine projectors (710A, 710B, 710C, 710D, 710E, 710F, 710G, 719H, 710I) can be organized into three non-conflicting projector groups (as indicated in the circles representing each projector, groups 1, 2, and 3).

[0031] Because the total collection time is limited by the conflicting projectors, a layout of N conflicting projectors (called a cluster) will require >= N groups. For example, a two-dimensional array of tiled projectors, regardless of array size, requires four or fewer cluster groups. Figure 8 The 4x6 array of projectors shown can be reduced to four non-overlapping clusters (group 1, group 2, group 3 and group 4) because projector 1,1 overlaps only with projectors 1,2 and 2,1, while projector 1,2 overlaps only with projectors 1,3 and 2,2, and so on.

[0032] Similarly, large dome displays also require only four or fewer clusters, such as Figure 9 As shown.

[0033] Turn Figure 10 According to one embodiment, a method for parallel collection of structured light patterns in a multi-projector system is proposed. This method utilizes graph coloring, a graph labeling algorithm by which projector nodes (called "colors") are assigned to nodes of a graph subject to certain constraints. In its simplest form, graph coloring is a way of coloring the vertices of a graph such that no two adjacent vertices have the same color (called vertex coloring).

[0034] At 1000, processor 150 sequentially turns projectors 110A, 110B, 110C, etc., of the multi-projector system 100 on / off. Sequential images are captured at 1010 by associated cameras, such as cameras 130A and 130B, and acquired within processor 150. At 1020, processor 150 creates projector / camera masks, such as masks 310, 310B, 310C, etc. As described above, steps 1000 and 1010 can be replaced by other steps based on prior knowledge of the location on the surface where the images from each projector will be projected, and prior knowledge of the relationship between at least one camera and the surface. At 1030, for each projector, processor 150 then uses the masks to create a graph of conflicting other projectors, such as... Figure 5A and 8 As shown in the diagram, when the projections of a projector (such as those seen from one or more cameras) overlap in a camera image, the projectors are marked as conflicting, and the projections cannot be unambiguous by the shape, color, timing, etc. of the projected pattern. Conflicting projectors are marked as "connected" to each other. All connections (and lack thereof) between projections form a graph, where each projector is a node / vertice of the graph, and each connection / conflict is an edge of the graph.

[0035] Because the time required to collect data from each projector can vary depending on the model, resolution, pattern, etc., knowledge of the collection time for each projector can be used to modify the chart (e.g., grouping projectors with similar speeds together).

[0036] This list can be created based on camera / projector conflicts and projector / projector conflicts; an example is shown in Tables 1 and 2:

[0037]

[0038] Based on the list created in step 1030, in step 1040, processor 150 uses, for example, a graph coloring algorithm to create clusters for simultaneous image acquisition. Many graph coloring algorithms and data structures are known in the art and can be used to create clusters in step 1040. For example, a greedy algorithm can be used, thereby... Figure 5B , 6A The vertices (i.e., nodes) of the projector diagrams shown in 6B, 7, and 9 are considered in a specific order, such as v1, ..., v n , where v i Assigned v1, ..., v n-1 Chinese v iThe minimum available color (i.e., cluster group) that the neighboring nodes do not use is selected, and new colors (groups) are added if necessary. Another approach uses graph decomposition, where the node with the highest connectivity (conflict) is selected and assigned to the first cluster group. Then, processor 150 searches for nodes outside the two projectors, assigning them to the same cluster group if there are no conflicts, and so on. A brute-force algorithm can also be used for a small number of projector assignments, where processor 150 determines every k-th assignment of k colors to n vertices and checks each assignment for validity. Finally, at 1050, images are acquired from the projectors of each cluster group by cameras (e.g., cameras 130A and 130B).

[0039] The invention has been described with respect to the exemplary embodiments described above. Other modifications and variations intended to be covered by the appended claims will be apparent to those skilled in the art. Furthermore, many modifications and variations will readily occur to those skilled in the art, and therefore it is not intended to limit the invention to the exact structures and operations shown and described, so that all suitable modifications and equivalents may be invoked and fall within the scope of the invention.

Claims

1. A method of collecting projected images on a surface in parallel in a multi-projector system having a plurality of projectors, comprising: creating one or more masks for limiting capture of projected images by at least one sensing device of each of the plurality of projectors to portions that do not overlap with projected images of other projections of the plurality of projectors; determining a graph of the plurality of projectors whose images conflict using the one or more masks, wherein the projected images are determined to conflict when they overlap in images captured by at least one sensing device, wherein each projector is a node of the graph, and wherein each conflict is an edge of the graph; creating clustered groups of the plurality of projectors whose images do not conflict from the graph using a graph coloring algorithm; modifying the graph using prior knowledge of relevant projector collection times such that projectors having approximately collection times are grouped into the same clustered group; and collecting images from the projectors in each of the clustered groups simultaneously by the at least one sensing device.

2. The method of claim 1, wherein, the graph coloring algorithm comprises at least one of a brute force algorithm, a node connectivity order based algorithm, a greedy algorithm, and a heuristic algorithm.

3. The method of claim 1, wherein, the images from the plurality of projectors are projected on a flat surface.

4. The method of claim 1, wherein, the images from the plurality of projectors are projected on a smooth surface.

5. The method of claim 4, wherein, the smooth surface comprises non-contiguous portions.

6. The method of claim 1, wherein, the images from the plurality of projectors are projected on a non-contiguous and / or non-smooth surface.

7. The method of claim 1, wherein, the images are structured light patterns.

8. The method of claim 1, wherein, the graph is determined by capturing the projected images on the surface via at least one camera.

9. The method of claim 8, further comprising separating a single camera image into images of each projector based on the masks.

10. The method of claim 1, wherein, the prior knowledge comprises projection design parameters.

11. The method of claim 1, wherein, the graph is determined via a previous iteration of the method.

10. The method of claim 1, wherein the plurality of projectors comprises at least 10 projectors.

Images