Information processing system, information processing program, and information processing method

Abstract

To provide an information processing system and the like that can execute calibration resistant to a noise such as design on a field.SOLUTION: An information processing system executes acquisition processing of acquiring correspondence information indicating the correspondence between the coordinate system of a projection device and the coordinate system of an imaging apparatus, on the basis of a pattern image and a picked-up image obtained by picking up an image of a field having design on which a pattern image having a plurality of types of polygons arranged thereon is projected.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to information processing for projecting an image by a projector. [Background technology]

[0002] Conventionally, a pattern image such as a chessboard pattern projected by a projector is captured by a camera, and calibration is performed on projector parameters and / or camera parameters based on the captured image (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-214206 Summary of the Invention [Problem to be solved by the invention]

[0004] When a design is drawn on the field, which is the projection area, there is a risk that the design will become noise and the pattern image projected on the field for calibration will not be accurately detected.

[0005] Therefore, an object of the present invention is to provide an information processing system and the like that can perform calibration that is resistant to noise from designs on a field when performing calibration by projecting a pattern image onto the field. [Means for solving the problem]

[0006] To achieve the above object, the following configuration examples can be given.

[0007] One example configuration is an information processing system that includes one or more processors, a projection device, an imaging device, and a field having a design on its surface, and the processor performs a projection process that causes the projection device to project a pattern image in which multiple types of polygons are arranged onto the surface of the field, an imaging process that causes the imaging device to capture the pattern image projected onto the surface of the field, and an acquisition process that acquires first correspondence information that indicates the correspondence between the coordinate system of the projection device and the coordinate system of the imaging device based on the pattern image and the captured image of the pattern image.

[0008] According to the above configuration example, by projecting a pattern image in which multiple types of polygons are arranged onto a field having a design on its surface, the projected pattern image can be detected more accurately even if the design of the field acts as noise. Therefore, it is possible to obtain a first correspondence relationship between the coordinate system of the projection device and the coordinate system of the imaging device, with the influence of noise suppressed.

[0009] As another example configuration, the processor may generate a luminance image based on the difference between the maximum and minimum luminance values ​​for each pixel across multiple frames in the captured image, and in the acquisition process, acquire the first correspondence information by matching features between the luminance image and the pattern image.

[0010] According to the above configuration example, the pattern image projected onto the field can be recognized as a luminance image based on the difference between the maximum and minimum luminance values. As a result, even if the design on the field and the pattern image projected onto the field overlap, the pattern image projected onto the field can be recognized while reducing the influence of the design on the field, thereby enabling highly accurate feature matching.

[0011] As another example of configuration, the pattern image may be an image in which a plurality of random polygons are arranged.

[0012] According to the above configuration example, the regularity of the pattern image is significantly reduced, so that the influence of noise can be suppressed more effectively.

[0013] As another configuration example, in the projection process, the processor may cause a projection device to sequentially project multiple pattern images onto the surface of the field, each of which has a different shape and / or position of each polygon to be placed; in the imaging process, the processor may cause an imaging device to capture the multiple pattern images sequentially projected onto the surface of the field; and in the acquisition process, the processor may acquire the first correspondence information based on the multiple pattern images and the captured image obtained by capturing the multiple pattern images.

[0014] According to the above configuration example, the first correspondence information is acquired using a plurality of pattern images, so that it is possible to suppress a decrease in accuracy of the first correspondence information.

[0015] In another configuration example, the field has a first design on its surface, and the processor causes the imaging device to capture an image of the first design on the field, detects the captured first design, and obtains second correspondence information indicating the correspondence between the coordinate system of the imaging device and the coordinate system of the field, and causes the projection device to project a predetermined image in accordance with the position of the field based on the first correspondence information and the second correspondence information.

[0016] According to the above configuration example, the second correspondence information indicating the correspondence relationship between the coordinate system of the imaging device and the coordinate system of the field can be acquired using the first design.

[0017] In another configuration example, a game may be played on the field using a combination of multiple types of card equipment, and the first design may be a design that indicates the combination.

[0018] According to the above configuration example, the second correspondence information indicating the correspondence relationship between the coordinate system of the imaging device and the coordinate system of the field can be acquired using the first design on the field indicating the combination of card accessories.

[0019] In another example configuration, the game may be played using card supplies placed on a field, the field having a second design on its surface, the second design being a design indicating the position on the field where the card supplies should be placed.

[0020] According to the above configuration example, when using a field that indicates to the player the position where to place card items using the second design, it is possible to obtain a first correspondence relationship between the coordinate system of the projection device and the coordinate system of the imaging device, while suppressing the influence of noise.

[0021] As another configuration example, the first correspondence information may be a projective transformation matrix that indicates the correspondence relationship between the coordinate system of the projection device and the coordinate system of the imaging device. [Effects of the Invention]

[0022] According to this embodiment, it is possible to provide an information processing system or the like that can perform calibration that is resistant to noise from designs on the field when a pattern image is projected onto the field to perform calibration. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram illustrating an example of a projection system and a state in which the projection system is arranged. [Figure 2] FIG. 10 is a diagram illustrating an example of a field [Figure 3] FIG. 10 is a diagram illustrating an example of a marker drawn on a field. [Figure 4] FIG. 1 is a diagram illustrating an example of a state in which an image is projected onto a field by a projection system. [Figure 5] FIG. 10 is a diagram for explaining a process of correcting distortion of a captured image by associating a field coordinate system with a camera coordinate system; [Figure 6] FIG. 10 is a diagram illustrating an example of a pattern image projected onto a field. [Figure 7] FIG. 10 is a diagram for explaining an example of a method for generating a pattern image. [Figure 8] FIG. 10 is a diagram for explaining an example of a method for generating a pattern image. [Figure 9] FIG. 10 is a diagram for explaining a process for associating a projector coordinate system with a camera coordinate system; [Figure 10] FIG. 10 is a diagram for explaining a process for associating a projector coordinate system with a camera coordinate system; [Figure 11] An example of a flowchart of the projection process according to this embodiment DETAILED DESCRIPTION OF THE INVENTION

[0024] An embodiment will be described below.

[0025] [Hardware configuration of information processing system]

[0026] An information processing system (sometimes referred to as a "projection system") according to an example of this embodiment will be described below. Fig. 1 is a diagram showing a projection system 100 according to this embodiment and an example of a state in which the projection system 100 is arranged. In Fig. 1, the projection system 100 includes a field 101, a projection device (sometimes referred to as a "projector") 102, an imaging device (sometimes referred to as a "camera") 103, and an information processing device 104.

[0027] The information processing device 104 includes a processor 105, a flash memory 106, and a dynamic random access memory (DRAM) 107. The information processing device 104 is connected to the projector 102 and the camera 103, and controls the projector 102 and the camera 103. The information processing device 104 is, for example, a personal computer.

[0028] Processor 105 is composed of one or more processors and executes information processing. Processor 105 executes information processing by executing an information processing program stored in a storage unit (an internal storage medium such as flash memory 106, and an external storage medium attached to information processing device 104, etc.). Flash memory 106 is a memory used mainly for storing various data (which may be programs) saved in information processing device 104. DRAM 107 is a memory used for temporarily storing various data used in information processing. Processor 105 executes information processing by appropriately reading and writing data between flash memory 106, DRAM 107, etc.

[0029] The field 101 has, for example, a rectangular board or sheet shape, and various designs are drawn on the surface. As will be described later, the players 11 and 12 place card items (sometimes simply referred to as "cards") on the surface of the field 101 to play a card game. In FIG. 1, the field 101 is placed on the table 10 with its surface facing up, and the players 11 and 12 face each other across the field 101, ready to play the card game.

[0030] The projector 102 is placed in a position facing the surface of the field 101, and projects various images onto the surface of the field 101 under the control of the information processing device 104. In FIG. 1, the projector 102 is placed on the ceiling of the room so as to overlook the field 101.

[0031] The camera 103 is placed at a position facing the surface of the field 101, and captures an image of the surface of the field 101 under the control of the information processing device 104. In FIG. 1, the camera 103 is placed on the ceiling of the room so as to look down on the field 101.

[0032] FIG. 2 is a diagram illustrating an example of the field 101. In this embodiment, as an example, a Hanafuda game is played on the field 101 using 48 square Hanafuda cards (a type of card equipment) in which two players 11 and 12 compete for points. The 48 Hanafuda cards have unique designs that are different from each other on the front and are solid black on the back. Note that in other embodiments, the game is not limited to the Hanafuda game, and a card game, mahjong game, or the like may also be played. In this case, a design corresponding to the game being played is drawn on the field.

[0033] As shown in FIG. 2, designs 110 to 117 are drawn on the surface of the field 101. Design 114 is a design of a plurality of rectangles indicating positions where cards from the player 12's hand are to be placed. Design 115 is a design of a plurality of rectangles indicating positions where cards from the player 11's hand are to be placed. Design 113 is a design of a plurality of rectangles indicating positions where cards from the tableau are to be placed.

[0034] Design 116 is a design of multiple squares indicating the positions where Hanafuda cards constituting a hand should be placed when the combination of the player 12's hand (hanafuda cards) placed on design 114 and the table cards (hanafuda cards) placed on design 113 forms a predetermined combination of a "yaku" (hand), and the player 12 wins points according to that hand. Design 117 is a design of multiple squares indicating the positions where Hanafuda cards constituting a hand should be placed when the combination of the player 11's hand (hanafuda cards) placed on design 115 and the table cards (hanafuda cards) placed on design 113 forms a predetermined combination of a "yaku" (hand), and the player 11 wins points according to that hand. Design 112 is a design of multiple squares indicating the positions where Hanafuda cards (deck cards) other than the Hanafuda cards placed on each of the above designs should be placed one on top of the other. Hanafuda cards are placed face up on designs 113 to 117, and Hanafuda cards are placed face down on design 112.

[0035] The design 110 is a design that shows the combination of Hanafuda cards that make up a winning hand to the player 12. The design 111 is the same design as the design 110, and is a design that shows the combination of Hanafuda cards that make up a winning hand to the player 11. As shown in FIG. 2, the designs 110 and 111 are drawn facing in opposite directions.

[0036] Figure 3 is an enlarged view of the designs 110 and 111 shown in Figure 2. As shown in Figure 3, the designs 110 and 111 depict multiple combinations of Hanafuda cards that make up a winning hand, which helps players who do not remember the winning hands. For convenience of illustration, Figure 3 depicts two types of winning hand designs that are made up of combinations of three Hanafuda cards, while the designs for other winning hands are omitted and represented by diagonal lines and the word "design" is written.

[0037] In the Hanafuda game of this embodiment, each player makes a hand using a combination of Hanafuda cards placed on the table and in their own hand, and competes by earning points according to the hand they make. A detailed explanation of the rules will be omitted. The projection system 100 supports the Hanafuda game by recognizing the Hanafuda cards that make up a hand among the Hanafuda cards on the table and in the player's hand, and showing the recognized hand to the player using a projected image (sometimes called a "support image") by the projector 102.

[0038] Fig. 4 is a diagram illustrating an example of a state in which a predetermined image showing Hanafuda cards constituting a winning combination is projected onto the field 101 by the projection system 100 of this embodiment. In Fig. 4, the Hanafuda cards are indicated by the reference numeral 130. Also, in Fig. 4, for convenience of illustration, the designs on the front sides of the Hanafuda cards are omitted and represented by diagonal lines, and the designs of the winning combination parts of the designs 110 and 111 are omitted and represented by diagonal lines. Note that in Figs. 2 and 9, the designs of the winning combination parts of the designs 110 and 111 are also omitted and represented by diagonal lines.

[0039] In Fig. 4, a combination of one of the Hanafuda cards on the table and one of the Hanafuda cards in the hand constitutes a winning hand. As shown in Fig. 4, a square frame-shaped support image 121 is projected around each of the Hanafuda cards that constitute the winning hand, indicating that the player can make a winning hand. In another embodiment, a linear support image connecting the projected frame-shaped support images 121 may be projected to indicate which Hanafuda cards (which Hanafuda cards have the projected support images 121 on them) can constitute a winning hand.

[0040] Here, in projection system 100, if the correspondence between the coordinate system of field 101 onto which supporting image 121 is projected (sometimes referred to as the "world coordinate system"), the coordinate system of projector 102 that projects supporting image 121 (sometimes referred to as the "projector coordinate system"), and the coordinate system of camera 103 (sometimes referred to as the "camera coordinate system") is not properly adjusted, supporting image 121 will not be projected with an appropriate position, orientation, shape, or size. Therefore, in this embodiment, the correspondence between the world coordinate system, projector coordinate system, and camera coordinate system is adjusted by the processing described below.

[0041] FIG. 5 is a diagram for explaining the process of correcting distortion in a captured image by associating a field coordinate system with a camera coordinate system. Note that FIG. 5 is a simplified illustration for convenience of illustration. FIG. 5(1) shows a world coordinate system in which a field 101 is placed (see FIG. 1, etc.). FIG. 5(2) is a captured image (camera coordinate system) obtained by capturing an image of the field 101 with a camera 103. As shown in FIG. 5, because the orientation of the camera 103 relative to the field 101 is misaligned, the shape of the field 101 reflected in the captured image is distorted in the camera coordinate system.

[0042] Here, the content, shape, size, and relative position of the designs 110 and 111 drawn on the field 101 are known (see FIGS. 2 and 3). Therefore, the processor 105 recognizes the designs 110 and 111 in the captured image of FIG. 5(2) by using the designs 110 and 111 as markers based on the known data (sometimes referred to as "marker data") previously stored in the flash memory 106 or the like. The processor 105 then calculates a projective transformation matrix that reflects the shapes, sizes, and relative positions of the designs 110 and 111 indicated by the marker data in the captured image in the camera coordinate system shown in FIG. 5(2). That is, the processor 105 calculates a projective transformation matrix (sometimes referred to as a "first projective transformation matrix") that defines the correspondence between the world coordinate system and the camera coordinate system. The processor 105 then corrects distortion in the captured image as shown in FIG. 5(3) by transforming the captured image using the first projective transformation matrix.

[0043] Next, we will explain the process of associating the projector coordinate system with the camera coordinate system in order to project the support image 121 without any deviation (see Figure 4) at a position on the field 101 recognized based on the captured image (see Figure 5(3)).

[0044] FIG. 6 shows an example of three pattern images sequentially projected onto the field 101 by the projector 102 to associate the projector coordinate system with the camera coordinate system. As will be described later, these pattern images are projected at positions overlapping with the design of the field 101 (see FIG. 2) (see FIGS. 2 and 9(1)). As shown in FIG. 6(1), the first pattern image is an image in which different polygons are arranged at the positions of the divided frames of a rectangle (sometimes referred to as a "first rectangle") that is divided vertically into 5 and horizontally into 8, with four points (points a, b, c, and d) in the projector coordinate system as vertices. As shown in FIG. 6(2), the second pattern image is an image in which different polygons are arranged at the positions of the divided frames of a rectangle (sometimes referred to as a "second rectangle") that is divided vertically into 6 and horizontally into 10, with four points (points a, b, c, and d) in the projector coordinate system as vertices. As shown in Figure 6(3), the third pattern image is an image in which different polygons are arranged at the positions of each division frame of a rectangle (sometimes called the "third rectangle") that is divided vertically into 8 and horizontally into 12, with four vertices (point a, point b, point c, point d) in the projector coordinate system. The coordinates of the four vertices (point a, point b, point c, point d) of the first rectangle, second rectangle, and third rectangle are the same coordinates in the projector coordinate system.

[0045] Note that the first rectangle, second rectangle, and third rectangle indicated by dashed lines in FIG. 6, as well as the divided frames of these rectangles, are shown for convenience of explanation and do not constitute the pattern image; polygons arranged in these rectangles are projected as the pattern image. Also, in FIG. 6 and other figures, for convenience of illustration, there are multiple polygons of the same shape, but the shapes of the polygons are different from each other. Also, in FIG. 6, polygons are arranged in frames divided from a rectangle, but polygons may also be arranged in frames divided from a square. Also, in FIG. 6, when polygons contact each other, the boundaries between the polygons are eliminated and the polygons are joined. Note that in other embodiments, polygons of the same shape and / or size (and even the same orientation) may exist in one pattern image.

[0046] Fig. 7 is a diagram for explaining a method for generating a pattern image. Fig. 7 explains a method for generating a first pattern image in which 40 polygons are arranged, but a second pattern image in which 60 polygons are arranged and a third pattern image in which 96 polygons are arranged are also generated in the same manner. Fig. 7 also explains a case in which a random polygon is generated in the upper left divided frame of the first rectangle, but random polygons are similarly generated for each of the other divided frames.

[0047] As shown in FIG. 7(2), processor 105 defines an ellipse inscribed in a division frame (see FIG. 7(1)) for generating a polygon. Next, processor 105 randomly determines the number N of vertices of the polygon to be generated from between 3 and 8. FIG. 7 illustrates a case where the number of vertices N is determined to be 6. Next, processor 105 determines the coordinates of each vertex of the polygon to be generated. FIG. 8 is an enlarged view of FIG. 7(2). A method for determining the coordinates of each vertex of the polygon to be generated will be described with reference to FIG. 8.

[0048] The coordinates of each vertex of the polygon to be generated are determined by polar coordinates (r, θk) where the center o of the ellipse is the pole, the distance from the pole o is r, and the argument θ [rad], as shown in Figure 8. The argument θk (k = 0, 1, ... N-1) of the polar coordinates is determined randomly within the range defined by [Equation 1].

[0049]

number

[0050] First, consider the case of determining the polar coordinates of the first vertex e. In this case, k = 0, and the argument θ0 of the polar coordinates of vertex e is randomly determined from the range where k = 0 in [Equation 1]. In FIG. 8, θ0 is determined to be π / 7. Next, as shown in FIG. 8, a straight line R connecting the center o (pole o) to the circumference of the ellipse is defined from among the straight lines formed by the coordinates that the polar coordinates (r, θ0) can take. Then, a value randomly determined from the range of 0.2 to 1.0 is multiplied by the length of the straight line R to determine the distance r from the pole o. In FIG. 8, the value obtained by multiplying the length of the straight line R by the randomly determined value 0.5 is determined to be the distance r from the pole o. In other words, half the length of the straight line R is determined to be the distance r from the pole o.

[0051] When determining the polar coordinate (r, θ1) of the second vertex f, k = 1, and the argument θ1 of the polar coordinate of vertex f is randomly determined from the range where k = 1 in [Equation 1]. The polar coordinate r of vertex f is also randomly determined using the method described above. The polar coordinate (r, θ2) of the third vertex g, the polar coordinate (r, θ3) of the fourth vertex h, the polar coordinate (r, θ4) of the fifth vertex i, and the polar coordinate (r, θ5) of the sixth vertex j are determined in the same manner.

[0052] Once the coordinates of the polygon's vertices have been determined in the above manner (see Figure 8 and Figure 7(2)), a polygon consisting of the determined vertices is defined as shown in Figure 7(3). Then, as shown in Figure 7(4), the defined polygon is fixed at the position of the center o of the ellipse and enlarged by an enlargement factor selected randomly from the values ​​of 1.2, 1.4, and 1.6. In Figure 7(4), the defined polygon is enlarged to 1.2 times its original size by an enlargement factor of 1.2. In this way, random polygons are generated for all the divided frames, and a first pattern image, a second pattern image, and a third pattern image are generated.

[0053] FIG. 9(1) is a diagram for explaining a state in which the projector 102 projects a first pattern image (see FIG. 6(1)) in a flashing manner onto the surface of the field 101. The projector 102 projects the pattern image in a flashing manner by, for example, projecting it for 0.5 seconds and then pausing the projection for 0.5 seconds, and repeating this process five times. Note that FIG. 9(1) shows the time point when the first pattern image is being projected.

[0054] FIG. 9(2) is a diagram showing a captured image (video) of the field 101 (see FIG. 9(1)) onto which the first pattern image is projected, captured by the camera 103. The captured image (video) shown in FIG. 9(2) shows the blinking of the first pattern image. Here, this captured image (video) is an image in which distortion of the captured image has been corrected using the first projective transformation matrix (first projective transformation matrix that defines the correspondence between the world coordinate system and the camera coordinate system) described with reference to FIG. 5.

[0055] FIG. 10(1) is a diagram showing a luminance image generated by calculating the difference between the maximum and minimum luminance values ​​for each pixel in a captured image (see FIG. 9(2)), which is a moving image, and visualizing the difference. As shown in FIG. 10(1), the luminance image ideally becomes an image of the shape of the first pattern image captured in the captured image. However, as shown in FIG. 9(2), the first pattern image is captured in the captured image overlapping with a design drawn on the field 101. This design may act as noise, causing the shape of the first pattern image in the luminance image to change partially. Furthermore, if a playing card or other object (such as a smartphone) is placed on the field 101, this may act as noise, causing the shape of the first pattern image in the luminance image to change partially.

[0056] Fig. 10(2) is a diagram showing feature matching performed between the first pattern image and the luminance image. As shown in Fig. 10(2), the processor 105 detects corresponding feature points between the first pattern image and the luminance image by performing feature matching between the first pattern image (projector coordinate system) and the luminance image (camera coordinate system). Note that the many straight lines in Fig. 10(2) are lines connecting corresponding feature points.

[0057] The processor 105 also performs a similar process using a second pattern image to detect corresponding feature points between the second pattern image and the luminance image of the second pattern image. The processor 105 also performs a similar process using a third pattern image to detect corresponding feature points between the third pattern image and the luminance image of the third pattern image.

[0058] Then, the processor 105 specifies a correspondence relationship between the coordinates of each feature point in the first pattern image and the coordinates of each feature point in the luminance image calculated using the first pattern image (i.e., a correspondence relationship between the projector coordinate system and the camera coordinate system). Similarly, the processor 105 specifies a correspondence relationship between the coordinates of each feature point in the second pattern image and the coordinates of each feature point in the luminance image calculated using the second pattern image, and also specifies a correspondence relationship between the coordinates of each feature point in the third pattern image and the coordinates of each feature point in the luminance image calculated using the third pattern image. That is, the processor 105 calculates a projective transformation matrix using each of the first, second, and third pattern images. Thereafter, the processor 105 calculates a projective transformation matrix (sometimes referred to as an "average projective transformation") by averaging the three specified projective transformation relationships, and calculates a projective transformation matrix (sometimes referred to as a "second projective transformation matrix") that defines the average projective transformation relationship. An example of a method for calculating the second projective transformation matrix will be specifically described below.

[0059] First, using a projective transformation matrix calculated using the first pattern image, the coordinates of four points (a1, b1, c1, d1) in the camera coordinate system corresponding to the coordinates of the four corner points (a, b, c, d; see FIG. 6) of the first pattern image (projector coordinate system) are calculated. Furthermore, using a projective transformation matrix calculated using the second pattern image, the coordinates of four points (a2, b2, c2, d2) in the camera coordinate system corresponding to the coordinates of the four corner points (a, b, c, d) of the second pattern image are calculated. Furthermore, using a projective transformation matrix calculated using the third pattern image, the coordinates of four points (a3, b3, c3, d3) in the camera coordinate system corresponding to the coordinates of the four corner points (a, b, c, d) of the third pattern image are calculated.

[0060] Furthermore, a reliability score s1 of the projective transformation calculated using the first pattern, a reliability score s2 of the projective transformation calculated using the second pattern, and a reliability score s3 of the projective transformation calculated using the third pattern are calculated. An example of a method for calculating the reliability scores will be described. The projective transformation matrix calculated using the first pattern is used to calculate coordinates of each feature point when the coordinates of each feature point included in the first pattern image (projector coordinate system; see FIGS. 6(1) and 10(2)) are transformed into the camera coordinate system. Then, the deviation (distance) between each coordinate in the calculated camera coordinate system and the coordinate of each feature point obtained from the luminance image of the first pattern image (camera coordinate system; see FIG. 10(1)) is calculated for each pair of feature points. Then, the average value of the calculated deviations (distances) is defined as the "projective transformation deviation value," and the reliability score s1 is calculated using the formula [reliability score = 1 / (1 + projective transformation deviation value)]. Similarly, the reliability score s2 and the reliability score s3 are calculated.

[0061] Then, using the calculated reliability scores, a "weighted average" is calculated for each of the coordinates of the four points in the camera coordinate system corresponding to the four corner points (a, b, c, d) of the pattern image. Specifically, the weighted average of a1, a2, and a3 is calculated as a4 = (s1 × a1 + s2 × a2 + s3 × a3) / (s1 + s2 + s3). The weighted average of b1, b2, and b3 is calculated as b4 = (s1 × b1 + s2 × b2 + s3 × b3) / (s1 + s2 + s3). The weighted average of c1, c2, and c3 is calculated as c4 = (s1 × c1 + s2 × c2 + s3 × c3) / (s1 + s2 + s3). Furthermore, the weighted average value of d1, d2, and d3 is calculated as d4=(s1×d1+s2×d2+s3×d3) / (s1+s2+s3). In other words, the average correspondence is calculated.

[0062] Then, a second projective transformation matrix is ​​calculated that transforms the coordinates of points a, b, c, and d in the projector coordinate system into the coordinates of points a4, b4, c4, and d4 in the camera coordinate system. By calculating the second projective transformation matrix using the weighted average value in this way, it is possible to calculate a highly reliable second projective transformation matrix.

[0063] Then, by converting and correcting the captured image using the calculated second projective transformation matrix, it is possible to project the supporting image 121 at an appropriate position without deviation, as described with reference to Fig. 4. In other words, it is possible to perform calibration for the camera coordinate system of the camera 103.

[0064] Here, a method of performing the above-described calibration of the camera coordinate system using a chessboard pattern with a regular shape as the pattern image, instead of a pattern image with randomly aligned polygons (see FIG. 6), is also conceivable. However, in this method, because the chessboard pattern has a regular shape, designs overlapping with the chessboard pattern in the projected image (such as designs drawn on the field 101 and designs of playing cards or other items placed on the field 101) may act as noise, causing the shape of the pattern image in the luminance image to change partially, which may significantly reduce the accuracy of feature matching. In this case, it may be impossible to properly calibrate the camera coordinate system, and it may not be possible to project the support image 121 at the appropriate position without misalignment (see FIG. 4). On the other hand, in this embodiment, a pattern image with randomly aligned polygons (see FIG. 6) is used as described above, allowing for accurate calibration that is resistant to noise.

[0065] Furthermore, the greater the number of polygons constituting the pattern image, the greater the number of feature points obtained by feature amount matching (see FIG. 6(3)), and therefore the accuracy of calibration improves. On the other hand, if the resolution of the pattern image projected onto the field 101 by the projector 102 is low and unclear, or if the resolution of the image captured by the camera 103 is low and unclear, the greater the number of polygons constituting the pattern image, the lower the accuracy of feature amount matching and the lower the accuracy of calibration. Therefore, in this embodiment, as described above, the second projective transformation matrix is ​​calculated using the first pattern image, the second pattern image, and the third pattern image (see FIG. 6), which have different numbers of polygons constituting the pattern images. This makes it possible to suppress a decrease in the accuracy of calibration.

[0066] [Details of information processing in this embodiment] Fig. 11 is an example of a flowchart of this embodiment. An example of information processing of this embodiment will be described in detail with reference to Fig. 11. When the projection system 100 is started and this information processing begins, the processing of Fig. 11 begins.

[0067] First, in step S101, the processor 105 controls the projector 102 to capture an image of the field 101, as described with reference to Figures 5(1) and 5(2). After that, the process proceeds to step S102.

[0068] In step S102, the processor 105 detects the markers (designs 110 and 111) in the captured image obtained in step S101 based on the marker data previously stored in the flash memory 106, etc., as described with reference to Fig. 5(2). Then, the process proceeds to step S103.

[0069] In step S103, the processor 105 calculates a first projective transformation matrix that defines the correspondence between the world coordinate system and the camera coordinate system based on the marker data and the markers in the captured image, as described with reference to Figures 5(2) and (3). Thereafter, the process proceeds to step S104.

[0070] In step S104, as described with reference to Fig. 9, processor 105 controls projector 102 to flash and project a pattern image (see Fig. 6) previously stored in flash memory 106 or the like onto field 101, and controls camera 103 to capture an image of field 101 onto which the pattern image is projected. Note that the pattern image may be generated by processor 105 when performing the processing of step S104. Thereafter, the processing proceeds to step S105.

[0071] In step S105, the processor 105 calculates a luminance image based on the captured image (video) acquired in step S104, as described with reference to Fig. 10(1). After that, the process proceeds to step S106.

[0072] In step S106, processor 105 determines whether or not the processes of steps S104 and S105 have been executed for three pattern images (first to third pattern images described with reference to FIG. 6). If the determination is YES, the process proceeds to step S107, and if the determination is NO, the process returns to step S104. Through the processes of steps S104 to S106, luminance images are calculated for the three pattern images.

[0073] In step S107, the processor 105 performs feature matching between the pattern image and the luminance image calculated from the pattern image, as described with reference to Fig. 10(2), to detect feature points in the luminance image. Then, the process proceeds to step S108.

[0074] In step S108, processor 105 determines whether or not the process of step S107 has been executed for three pattern images (see the first to third pattern images described with reference to FIG. 6). If the determination is YES, the process proceeds to step S109, and if the determination is NO, the process returns to step S107. By the processes of steps S107 and S108, feature points are detected for each of the three luminance images. Thereafter, the process proceeds to step S109.

[0075] In step S109, the processor 105 identifies three correspondence relationships between the projector coordinate system and the camera coordinate system, as already described, and calculates the average correspondence relationship of the identified three correspondence relationships. Then, the process proceeds to step S110.

[0076] In step S110, the processor 105 calculates a second homography matrix that defines the average correspondence between the calculated camera coordinate system and the projector coordinate system, as described in the description of Figure 10(2). After that, the process proceeds to step S111.

[0077] In step S111, the processor 105 executes a projection process using the first projective transformation matrix calculated in step S103 and the second projective transformation matrix calculated in step S109, and projects the support image 121 onto the field 101 to support the Hanafuda game, for example, as described with reference to Fig. 4. Thereafter, when the Hanafuda game ends or the projection system 100 is stopped, this information processing ends.

[0078] As described above, according to this embodiment, a pattern image (see FIG. 6) in which random polygons are aligned is used, so it is possible to prevent a significant decrease in the accuracy of feature matching due to the influence of the design drawn on the field 101, etc., and to perform accurate calibration that is resistant to noise. Furthermore, as described above, according to this embodiment, the second projective transformation matrix is ​​calculated using the first pattern image, the second pattern image, and the third pattern image (see FIG. 6), which have different numbers of polygons constituting the pattern images, so it is possible to suppress a decrease in the accuracy of the calibration.

[0079] [Variations] In the above-described embodiment, a case has been described in which a series of steps related to information processing is executed by a single information processing device. In other embodiments, the series of steps may be executed in a system consisting of multiple information processing devices. For example, in a system including a terminal device and a server device capable of communicating with the terminal device via a network, some of the steps may be executed by the server device. Furthermore, in a system including a terminal device and a server device capable of communicating with the terminal device via a network, main steps of the series of steps may be executed by the server device, and some of the steps may be executed by the terminal device. Furthermore, in the above-described system, the server system may be composed of multiple information processing devices, and the processes to be executed on the server side may be shared and executed by the multiple information processing devices.

[0080] Although the present embodiment and its modifications have been described above, these descriptions are merely illustrative in every respect and are not intended to limit the scope of the present embodiment and its modifications. It goes without saying that various improvements and modifications can be made to the present embodiment and its modifications. [Explanation of symbols]

[0081] 100 Projection System 101 Field 102 Projection device 103 Imaging device 104 Information processing equipment

Claims

1. one or more processors; A projection device; An imaging device; a field having a design on its surface; The processor: a projection process for projecting a pattern image in which a plurality of types of polygons are arranged onto the surface of the field by the projection device; an imaging process for causing the imaging device to capture the pattern image projected onto the surface of the field; and executing an acquisition process to acquire first correspondence information indicating a correspondence relationship between the coordinate system of the projection device and the coordinate system of the imaging device based on the pattern image and a captured image obtained by capturing the pattern image.

2. The processor: generating a luminance image based on a difference between a maximum value and a minimum value of luminance for each pixel across a plurality of frames in the captured image; The information processing system according to claim 1 , wherein the first correspondence information is acquired by matching features between the luminance image and the pattern image in the acquisition process.

3. The information processing system according to claim 1 , wherein the pattern image is an image in which a plurality of random polygons are arranged.

4. The processor: In the projection process, the projection device sequentially projects a plurality of the pattern images, each of which has a different shape and / or position of each polygon to be placed, onto the surface of the field; In the imaging process, the imaging device is caused to capture a plurality of the pattern images projected in sequence onto the surface of the field; The information processing system according to claim 1 , wherein the first correspondence information is acquired in the acquisition process based on a plurality of the pattern images and a captured image of the plurality of the pattern images.

5. the field having a first design on the surface; The processor: causing the imaging device to capture an image of the first design on the field; By detecting the captured first design, second correspondence information indicating a correspondence relationship between a coordinate system of the imaging device and a coordinate system of the field is acquired; The information processing system according to claim 1 , wherein the projection device projects a predetermined image in accordance with a position of the field based on the first correspondence information and the second correspondence information.

6. A game is played on the field using a combination of multiple types of card equipment, The information processing system according to claim 5 , wherein the first design is a design that indicates the combination.

7. the game is played using the card equipment placed on the field, the field having a second design on the surface; 7. The information processing system according to claim 6, wherein the second design indicates a position on the field where the card equipment is to be placed.

8. The information processing system according to claim 1 , wherein the first correspondence information is a projective transformation matrix that indicates a correspondence relationship between a coordinate system of the projection device and a coordinate system of the image capture device.

9. One or more processors of the information processing device a projection process for projecting a pattern image in which a plurality of types of polygons are arranged onto a surface of a field having a design on the surface of the field, using a projection device; an imaging process for causing an imaging device to capture the pattern image projected onto the surface of the field; and an acquisition process for acquiring first correspondence information indicating a correspondence relationship between the coordinate system of the projection device and the coordinate system of the imaging device based on the pattern image and a captured image of the pattern image.

10. the processor, generating a luminance image based on a difference between a maximum value and a minimum value of luminance for each pixel across a plurality of frames in the captured image; The information processing program according to claim 9 , wherein the acquisition process acquires the first correspondence information by matching feature amounts between the luminance image and the pattern image.

11. The information processing program according to claim 9 or 10, wherein the pattern image is an image in which a plurality of random polygons are arranged.

12. the processor, In the projection process, the projection device sequentially projects a plurality of the pattern images, each of which has a different shape and / or position of each polygon to be placed, onto the surface of the field; In the imaging process, the imaging device is caused to capture a plurality of the pattern images projected in sequence onto the surface of the field; The information processing program according to claim 9 , wherein the acquisition process causes the first correspondence information to be acquired based on a plurality of the pattern images and a captured image of the plurality of the pattern images.

13. the field having a first design on the surface; the processor, causing the imaging device to capture an image of the first design on the field; detecting the captured first design to obtain second correspondence information indicating a correspondence relationship between a coordinate system of the imaging device and a coordinate system of the field; The information processing program according to claim 9 , further comprising causing the projection device to project a predetermined image in accordance with a position of the field based on the first correspondence information and the second correspondence information.

14. A game is played on the field using a combination of multiple types of card equipment, The information processing program according to claim 13 , wherein the first design is a design that indicates the combination.

15. the game is played using the card equipment placed on the field, the field having a second design on the surface; The information processing program according to claim 14 , wherein the second design is a design indicating a position on the field where the card equipment is to be placed.

16. The information processing program according to claim 9 , wherein the first correspondence information is a projective transformation matrix indicating a correspondence relationship between a coordinate system of the projection device and a coordinate system of the imaging device.

17. An information processing method executed by one or more processors of an information processing device, a projection process for projecting a pattern image in which a plurality of types of polygons are arranged onto a surface of a field having a design on the surface of the field, using a projection device; an imaging process for causing an imaging device to capture the pattern image projected onto the surface of the field; and executing an acquisition process to acquire first correspondence information indicating a correspondence relationship between the coordinate system of the projection device and the coordinate system of the imaging device based on the pattern image and a captured image obtained by capturing the pattern image.

Images